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Abstract 

In  today’s  world  of  advanced  computing  power  at  the  fingertips  of  any  user, 
computer  security  should  be  a  primary  concern.  Information  is  power  and  this  power 
is  within  the  computer  system.  If  the  information  within  computer  systems  cannot 
be  trusted  then  the  power  that  comes  from  such  information  cannot  be  properly 
used.  Rootkits  are  software  programs  that  are  designed  to  establish  and  maintain 
an  environment  in  which  malware  may  hide  on  a  computer  system  after  successful 
compromise  of  that  computer  system.  Rootkits  cut  at  the  very  foundation  of  the  trust 
in  information  and  subsequent  power. 

This  thesis  examines  rootkit  hiding  techniques,  rootkit  finding  techniques  and 
develops  attack  trees  and  defense  trees  to  identify  deficiencies  in  detection  and  further 
increase  the  trust  in  information  systems.  The  developed  attack  and  defense  trees 
identified  that  enumeration  is  not  sufficient  to  defend  against  rootkits.  A  developed 
classification  of  rootkits  helps  fill  the  gaps  in  enumeration  of  rootkit  techniques  and 
gives  direction  for  further  detection  development.  By  fully  understanding  what  areas 
need  to  be  addressed  in  detection,  better  and  more  complete  tools  will  come  to  fruition. 
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A  Study  of  Rootkit  Stealth  Techniques 
and  Associated  Detection  Methods 

I.  Introduction 

1 . 1  Motivation 

Computer  networks  and  their  associated  devices  have  been  under  attack  by 
hackers  for  some  time,  and  defenses  have  been  developed  to  protect  against  a  myriad 
of  attacks.  However,  in  recent  years  a  powerful  new  class  of  software  called  rootkits 
and  also  known  as  stealth  technology  has  been  developed. 

Rootkits  are  software  programs  that  establish  and  maintain  an  environment 
in  which  malware  can  hide  on  a  computer  system  after  successful  compromise  of 
that  computer  system.  Since  the  inception  of  rootkits  in  the  1980s  [8],  rootkits  have 
improved  in  both  their  techniques  and  ability  to  protect  hackers  and  their  tools  from 
being  discovered.  McAfee(R)  Avert(R)  Labs  indicates  that  over  the  last  three  years 
the  “incident  rate  of  stealth  technology  has  increased  by  more  than  600  percent  [34, 
35]”,  and  “The  number  of  rootkits  submitted  ...  in  2006  compared  to  the  first  quarter 
of  2005  increased  by  nearly  700%.  The  number  of  Windows-based  stealth  components 
dominate  the  landscape,  with  an  increase  of  2300%  from  2001  to  2005  [34,35].” 

There  is  often  ample  time  between  the  discovery  of  vulnerabilities  and  patching 
of  that  vulnerability  to  install  a  rootkit.  Installation  of  a  rootkit  compounds  the  effects 
of  the  original  compromise  because  a  rootkit  creates  and  maintains  an  environment 
in  which  a  software  entity  may  hide  itself  and  its  effects  thus  making  the  original 
compromise  much  more  difficult  to  detect  and  remove. 

There  are  many  ways  to  mitigate  threats  to  a  computer  system  such  as  fire¬ 
walls,  Intrusion  Detection  Systems  (IDS)  and  Instrusion  Prevention  Systems  (IPS). 
However,  because  of  the  time  gap  between  vulnerabilities  and  patches,  the  increasing 
capabilities  of  exploits,  and  the  large  increase  in  numbers  of  rootkits  computers  are 
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still  very  vulnerable.  Thus,  it  is  vital  to  understand  how  rootkits  hide  and  how  they 
are  found  in  order  to  build  a  rootkit  attack  tree  that  can  identify  holes  in  detection 
techniques. 

1.2  Overview 

Rootkits  create  and  maintain  an  environment  for  attack  tools,  such  that  a  user 
does  not  know  of  their  presence  on  a  compromised  machine.  However,  the  rootkit  does 
not  gain  access  to  the  machine,  rather,  it  maintains  the  access.  In  this  research  Rootkit 
Hiding  Techniques  (RHTs)are  explored  as  well  as  the  Rootkit  Detection  Techniques 
(RDTs)  that  are  being  used  by  some  currently  available  tools.  We  analyze  these 
hiding  and  discovery  techniques  to  identify  deficiencies  in  detection. 

Attack  trees  for  rootkits  are  developed  which  give  a  high  level  overview  of  impor¬ 
tant  areas  to  defend  in  computer  systems.  An  experiment  in  detection  is  conducted 
using  a  rootkit  to  find  rootkits.  Although  operating  systems  other  than  Windows(R) 
are  discussed,  this  research  focuses  on  Windows(R)  rootkits  for  two  reasons:  First, 
operating  systems  differ  in  names  for  various  tables,  in  functionality,  and  in  stability 
but  the  concepts  are  roughly  the  same  (i.e. ,  each  has  hardware  interface(s),  sched¬ 
uler,  kernel,  libraries,  system  calls,  and  distinctions  between  user  and  kernel  level 
permissions)  meaning  the  same  general  conclusions  and  attack  trees  can  be  shared 
with  minor  modifications  to  suit  each  system  (cf.,  Appendicies  A  and  B.)  Second,  ac¬ 
cording  to  McAffee(R)  “The  share  of  Linux-based  techniques  has  gone  from  a  high  of 
roughly  72  percent  of  all  malware  stealth  components  in  2001  to  a  negligible  number 
in  2005,  while  the  number  of  Windows(R)-based  stealth  components  has  increased  by 
2,300  percent  in  the  same  time  period  [35].” 

1.3  Research  Statement 

In  this  research  we  study  the  methods  by  which  rootkits  hide  and  also  how  they 
can  be  detected.  These  hiding  and  finding  techniques  are  used  to  create  attack  trees 
from  which  deficiencies  in  current  detection  techniques  can  be  identified,  identified 
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deficiencies  can  be  used  to  increase  the  effectiveness  of  computer  system  defenses. 
This  effort  includes  examples  of  current  rootkits  and  rootkit  detectors. 

1.4  Thesis  Organization 

This  chapter  gives  motivation,  an  overview,  and  a  research  statement  as  well  as 
the  organization  of  the  thesis.  Chapter  Two  contains  a  literature  review  that  encom¬ 
passes  operating  systems,  classification  systems,  trusted  computing,  exploits  versus 
patches,  and  gives  a  general  background  on  rootkits.  Chapter  Three  describes  vari¬ 
ous  hiding  techniques  as  well  as  examples  of  these  techniques.  The  hiding  techniques 
are  followed  by  various  finding  techniques  used  by  RDTs.  Chapter  Four  explains  the 
research  findings  and  describes  the  System  Under  Test  (SUT)  as  well  as  experimental 
results.  Finally,  Chapter  Five  presents  conclusions  and  recommendations  for  future 
research. 
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II.  Literature  Review 


2. 1  Background 

Attacks  against  computer  systems  have  a  wide  range  of  effects  and  purpose. 
Some  attacks  are  meant  to  be  malicious  in  various  degrees,  while  others  are  simply 
for  curiosity.  Regardless  of  the  intent  of  an  attack,  we  require  the  ability  to  defend 
our  computer  systems  and  allow/deny  access  to  our  computer  systems  as  we  see  fit, 
not  as  an  intruder  may  see  fit.  In  attempt  to  protect  and  study  the  defense  of  our 
computer  systems  many  classification  systems  or  taxonomies  have  been  developed 
and  are  discussed  in  Section  2.3. 

Computer  attacks  may  have  a  long  period  of  time  in  which  to  occur  because  of 
the  length  of  time  between  discovery  of  a  vulnerability  (a  software  flaw  that  allows 
other  than  intended  operations  to  occur)  and  the  application  of  a  patch  (a  piece  of 
code  used  to  replace/fix  software  vulnerabilities),  which  is  further  explained  in  Sec¬ 
tion  2.5  of  this  thesis.  If  vulnerabilities  are  left  unchecked  and  a  threat  (something 
willing  to  take  advantage  of  the  vulnerability)  exists  then  our  computer  systems  are 
at  risk.  This  risk  ranges  from  exposure  (the  simple  loss  of  data)  to  the  complete 
loss  of  control  over  a  machine.  The  next  step,  once  a  successful  computer  attack  has 
occurred,  is  that  of  maintaining  control;  this  control  can  be  maintained  by  rootkit 
software.  Just  as  a  country  does  not  or  should  not  fight  a  battle  and  ignore  the 
war,  a  computer  attack  is  simply  the  beginning  (a  battle)  to  lay  the  ground  work 
for  maintaining  control  over  the  computer  asset  (the  war).  As  noted  by  Anton  Chu- 
vakin  of  iDEFENSE  Labs,  “Rootkits  are  automated  software  packages  to  setup  and 
maintain  an  environment  on  a  compromised  machine.”  According  to  Chuvakin,  the 
first  evidence  in  the  public  domain  of  Rootkits  was  discovered  in  the  mid  1990s  [15]. 
Rootkits  are  explained  further  in  Section  2.6,  of  this  thesis.  This  thesis  will  seek  to 
help  identify  rootkit  stealth  techniques  and  identify  deficiencies  in  rootkit  detection 
techniques.  By  identifying  deficiencies  in  detection  we  can  learn  to  better  detect  and 
subsequently  remove  rootkits.  The  subsequent  sections  of  this  chapter  will  give  the 
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reader  a  background  in  operating  systems,  classification  systems,  trusted  computing, 
exploits  vs  patches,  and  rootkits. 

2.2  Operating  Systems 

In  order  to  understand  how  a  rootkit  might  install  itself  and  hide  in  a  computer 
we  must  understand  a  computer  from  the  architecture  level.  Silberschatz,  Gagne,  and 
Galvin  in  their  book  Operating  System  Concepts,  divide  a  computer  system  into  four 
main  areas:  hardware,  the  operating  system,  application  software,  and  users  using 
Figure  2.1  to  illustrate  [59]. 


Figure  2.1:  Abstract  view  of  the  components  of  a  computer 

system  [59]. 


The  user  of  a  computer  system  uses  hardware  input/output  devices  such  as  the 
keyboard  and  mouse  in  order  to  start  desired  tasks.  Those  hardware  I/O  devices 
interact  with  application  software  in  order  to  accomplish  the  desired  tasks.  The 
operating  system  is  the  intermediary  between  the  hardware,  user,  and  application 
software  which  acts  as  a  manager  between  the  user,  the  hardware,  and  the  application 
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software  of  a  computer  system  [59,65].  A  good  example  used  by  Silberschatz  et  al, 
is  that  the  operating  system  is  “Like  a  government,  it  performs  no  useful  function  by 
itself.  It  simply  provides  an  environment  within  which  other  programs  can  do  useful 
work”  [59], 

If  we  look  at  Figure  2.1,  we  notice  that  there  are  a  few  layers  within  the  diagram 
in  which  a  rootkit  could  insert  itself,  namely:  at  the  application  level,  the  operating 
system  level,  and  at  the  hardware  level.  Although  inserting  rootkits  at  the  hardware 
level  may  be  feasible,  as  documented  by  John  Heasman  in  his  paper  Implementing  and 
Detecting  a  PCI  Rootkit  [28],  for  this  thesis,  we  will  focus  on  the  first  two  levels,  paying 
special  attention  to  the  operating  system  level  with  its  many  sub-levels.  Within  each 
operating  system  is  a  structure  called  a  kernel  which,  according  to  McKusick  and 
Nevillc-Neil  in  their  book  The  Design  and  Implementation  of  the  FreeBSD  Operating 
System,  “...is  a  small  nucleus  of  software  that  provides  only  the  minimal  facilities 
necessary  for  implementing  additional  operating-system  services”  [43].  The  size  and 
functionality  of  kernels  have  increased  over  the  years  but  the  intent  remains  the  same, 
such  that  the  kernel  controls  what  and  when  each  thread,  process,  or  task  is  run  and 
for  how  long.  The  following  subsections  will  give  the  background  needed  for  the 
Windows,  Linux,  and  BSD  OSs.  Furthermore,  these  sections  will  show  what  each  of 
the  main  divisions  are  in  each  OS.  However,  as  will  be  seen,  the  operating  systems 
are  very  similar  at  least  from  a  high  level  component  view  and  attack  perspective. 

Not  all  of  the  tables  or  important  structures  will  be  named  equivalently  be¬ 
tween  operating  systems  but  the  underlying  architecture  of  how  an  operating  system 
behaves  when  an  input /interrupt  occurs  is  similar.  Max  Bruning  states  in  his  ar¬ 
ticle  A  comparison  of  Solaris,  Linux,  and  FreeBSD  that,  “Once  you  get  past  the 
different  naming  conventions,  each  OS  takes  fairly  similar  paths  toward  implementing 
the  different  concepts”  [6].  For  example,  in  Linux  or  Windows,  when  an  interrupt 
is  triggered,  the  interrupt  handler  takes  over.  The  interrupt  handler  will  then  check 
an  interrupt  descriptor  table  (IDT)  to  know  how  to  handle  the  particular  kind  of 
interrupt /system  call  request,  the  system  call  table  is  checked  and  then  control  jumps 
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to  the  address  found.  Control  of  this  process  can  be  usurped  either  by  changing  the 
interrupt  handler  so  that  it  uses  a  completely  new  IDT,  or  by  changing  addresses  in 
the  IDT  [64].  Execution  of  a  system  call  in  a  UNIX-like  operating  systems  (which 
parallels  that  of  a  Windows  OS)  is  shown  in  Figure  2.2. 

call  open() 

user  mode  flow  of  execution 

kernel  mode _ _ 


choose  interrupt  handler 

choose  system  call 

sys_open() 

4 

_ i  i  _ 1/ _ 

interrupt  descriptor  table  (IDT )  syscall  table 

Figure  2.2:  UNIX-like  OS:  System  call  flow  [7]. 

2.2.1  Windows  OS.  According  to  Silberschatz  et  al,  Windows  was  designed 
for  “...security,  reliability,  Windows  and  POSIX  application  compatibility,  high  per¬ 
formance,  extensibility,  portability,  and  international  support”  [60].  As  can  be  seen  in 
Figure  2.3,  the  Windows  OS  was  created  in  many  modules.  The  main  areas  of  which 
to  take  note  are  the  Hardware-Abstraction  Layer,  the  Kernel,  the  executive,  and  the 
usermode  subsystems.  The  Kernel  is  the  part  of  the  operating  system  that  decides 
what,  when,  and  for  how  long  each  thread  is  going  to  run  based  on  parameters  that 
it  receives  as  to  priority  [60]. 

In  the  Windows  OS  we  find  rootkits  of  three  main  types,  namely  patching 
(replacement  of  code  sections),  hooking  (altering  execution  paths),  and  data  structure 
manipulation  (altering  data  structures).  Windows  much  like  any  operating  system  is 
subject  to  attacks  via  patching  because  of  the  need  to  allow  programs  to  run  with 
varying  permissions  which  can  be  granted  by  an  unknowing  user.  However,  steps  have 
been  taken  to  deter  this  concept  of  patching  by  concepts  such  as  digital  signing  which 
can  verify  the  integrity  of  applications. 
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Figure  2.3:  Block  Diagram  of  Windows  XP  OS  [59]. 


Windows  is  also  subject  to  hooking,  in  part  because  this  functionality  is  allowed 
(given  the  appropriate  permissions)  in  order  to  monitor  other  processes.  Finally,  Win¬ 
dows  is  also  subject  to  data  structure  manipulation  because,  as  with  other  operating 
systems,  there  are  data  structures  in  software  which  can  be  modified.  These  prob¬ 
lems  are  not  easily  addressed  because  if  we  completely  removed  the  ability  to  alter 
applications,  execution  paths  and  structures,  we  would  remove  desired  functionality 
and  upgrade  ability. 


2.2.2  Linux  &  FreeBSD  OSs.  Although  there  are  differences  in  implemen¬ 
tation  and  some  functionality  between  Linux  and  FreeBSD,  their  basic  architecture  is 
virtually  the  same  and  grows  closer  together  each  time  there  is  a  development  in  one 
or  the  other,  due  to  cross  pollination  of  ideas  and  sharing.  According  to  Silberschatz 
et  al,  Linux  was  designed  for  “...speed,  efficiency,  and  standardization”  [59].  Accord¬ 
ing  to  McKusick  et  al,  “the  FreeBSD  kernel  provides  four  basic  facilities:  processes,  a 
filesystem,  communications,  and  system  startup”  [43].  As  can  be  seen  in  Figure  2.4, 


the  Linux  OS,  much  like  Windows,  was  created  in  many  modules.  The  similarities 
between  Windows  and  Linux  can  been  seen  such  that  both  systems  have  a  kernel 
which  is  surrounded  by  supporting  modules  to  interact  with  higher  level  users  and 
lower  level  hardware.  However,  a  difference  between  Windows  and  Linux  is  in  regard 
to  processes  and  threads.  Windows  uses  threads  as  its  fundamental  unit  of  execution 
while  Linux  does  not  really  distinguish  between  processes  and  threads,  rather,  Linux 
generally  refers  to  both  as  tasks  [59]. 


system- 
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programs 


user 

processes 
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utility 

programs 
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system  shared  libraries 
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Figure  2.4:  Block  Diagram  of  Linux  OS  [59]. 


One  of  the  first  techniques  used  by  Linux  rootkits  was  patching  which  is  the 
replacement  or  modification  of  data,  hies,  binaries,  etc.  An  example  of  such  is  the 
rootkit  TOrn  which  “replaces  login ,  ifconfig,  ps ,  du,  Is,  netstat,  in. fingered,  find  and 
top ”  [7]  with  modified  versions  of  the  same  hies.  These  modified  versions  then  work  to 
hide  information  from  the  user.  For  example,  the  Is  command  shows  a  list  of  what  hies 
are  present  in  the  current  directory.  The  modified  Is  command  would  hlter  from  view 
any  hies  specihed  by  the  rootkit.  One  of  the  most  common  techniques  for  insertion  of 
a  rootkit  in  Linux  and  BSD  is  through  a  kernel  loadable  module  which  uses  all  three 
rootkit  techniques.  A  loadable  kernel  module  allows  any  information  processed  by  the 
system  to  be  modified  [7].  This  runtime  insertion  of  malicious  code  into  the  kernel  can 
be  deterred  by  turning  oh  the  ability  to  load  kernel  modules.  However,  as  discussed 
previously  this  can  remove  functionality  that  was  used  for  legitimate  purposes. 


2.2.3  Prevention.  Certainly  there  are  conhgurations  that  make  each  of  the 
various  operating  systems  more  secure  such  as  restricting  the  permissions  a  user  has 
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on  their  account  (ie,  user  account  instead  of  administrator  or  root).  In  the  various 
UNIX  variants,  removing  the  ability  to  load  kernel  modules  by  turning  off  the  func¬ 
tionality  slows  down  the  main  class  of  rootkits  because,  as  discussed  previously,  the 
ability  to  load  malicious  code  would  then  need  to  occur  in  a  different  way  such  as 
loading  through  /dev/kmem.  The  device  hies  /dev /mem  and  /dev/kmem  “allow  the 
root  user  to  arbitrarily  access  the  contents  of  physical  memory  and  kernel  memory, 
respectively  [21].”  Allowing  a  root  user  to  access  physical  and  kernel  memory  gives 
all  the  power  desired  to  rootkit  authors  that  have  obtained  root  permissions  through 
some  vulnerability. 

Mandatory  access  controls  (“When  a  system  mechanism  controls  access  to  an 
object  and  an  individual  user  cannot  alter  that  access”  [5])  can  also  be  added  in 
some  UNIX  variants  such  as  Trusted  Debian  (aka  Adamantix)  [1],  SE  Linux  [46],  and 
Trusted  BSD  [68].  Other  functions  also  exist  to  help  secure  a  system,  FreeBSD  for 
example,  also  has  a  function  called  “jail”  which  restricts  the  user  of  that  jail  to  its 
own  associated  “processes,  hies,  and  network  [43].”  There  are  also  many  software  and 
hardware  suites  that  help  keep  our  computer  systems  relatively  clean.  Firewalls,  Host 
Intrusion  Detection  Systems  (HIDS),  Network  Intrusion  Detection  Systems  (NIDS), 
Anti-Virus(AV),  and  various  Rootkit  Detectors.  Another  example  includes  the  hie 
hags  and  four  security  levels  in  BSD  in  order  to  further  secure  the  kernel.  These 
security  levels  are: 

“-1.  Permanently  insecure  mode:  Kernel  runs  at  secure  level  0  at  all  times. 

0.  Insecure  mode:  File  hags  may  be  set  and  reset  and  devices  may  be  read 
from  or  written  to  according  to  their  permissions 

1.  Secure  mode:  Superuser-setable  hie  hags  cannot  be  turned  oh.  Device 
hies  for  system  memory  / dev/mem ,  /dev/kmem  and  for  mounted  filesys¬ 
tems  are  read-only. 

2.  Highly  secure  mode:  Device  hies  for  filesystems  are  always  read-only, 
whether  they  are  mounted  or  not.  Firewall  rules  may  not  be  changed”  [70]. 


10 


2.3  Classifications 

Many  classifications  have  been  created  in  order  to  understand  and  protect  our 
networks.  Some  of  the  classifications  that  have  been  used  include:  achieved  access, 
automation,  attack  type ,  and  attack  tree. 

2.3.1  Achieved  Access.  In  Table  2.1,  we  see  Incident  Categories  from  Air 
Force  Instruction  33-138,  which  shows  the  categories  in  which  incidents  fall  based 
on  the  level  of  achieved  access  by  the  attacker.  Although  this  type  of  classification 
system  is  useful  in  categorizing  what  has  occurred  in  order  to  react  it  does  not  cover 
how  it  occurred  in  order  to  defend. 

2.3.2  Automation.  Further  classifications  can  be  found,  such  as  classifying 
by  degree  of  automation  used  in  the  attack:  Manual,  Semi-Automatic,  and  Automatic. 
Manual  attacks  are  as  the  name  implies,  where  the  human  attacker  scans,  breaks  into, 
installs,  and  then  uses  some  attack  tool.  Semi-automatic  attacks,  use  scripts  to  do  the 
scanning,  breaking,  and  installation,  which  is  then  followed  by  the  manual  instructions 
to  use  the  installed  tools.  Attacks  are  classified  as  automatic  when  the  human-in-the- 
loop  simply  starts  the  attack  script  and  the  scanning,  breaking,  installation,  and 
exploitation  all  occur  without  intervention  [37]. 

2.3.3  Attack  Type.  Attack  types  are  simply  descriptive  names  given  to 
attacks  such  as  IP  Spoofing,  Source  Routing,  Routing  Table  Corruption,  Denial  Of 
Service  Attacks  (DOS),  Smurf  Attacks,  Land  Attack,  Xmas  Tree  Attack,  Teardrop, 
TCP/UDP  Diag  Services  Attacks,  Ping  of  Death,  SYN  Flood,  and  Session  Hijack¬ 
ing  [42], 

2.3.4  Attack  Tree.  An  attack  tree  is  simply  the  graphical  representation  of 
how  an  attack  reaches  its  goal.  Bruce  Schneier,  the  CTO  of  Counterpane  Internet 
Security,  describes  attack  trees  as  follows:  “Attack  trees  provide  a  formal,  methodical 
way  of  describing  the  security  of  systems,  based  on  varying  attacks.  Basically,  you 
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Tabic  2.1:  Incident  Categories  (AFI  33-138)  [2] 


Category 

Description 

/ 

Root-Level  Intrusion:  An  unauthorized  person  gained  root-level 
access/privileges  on  an  Air  Force  computer/information  system/network 
device. 

II 

User-Level  Intrusion:  An  unauthorized  person  gained  user-level 
privileges  on  an  Air  Force  computer/information  system/network 
device. 

III 

Attempted  Access:  An  unauthorized  person  specifically  targeted  a 
service/vulnerability  on  an  Air  Force  computer /information 
system/network  device  in  an  attempt  to  gain  unauthorized  or  increased 
access/privilcges,  but  was  denied  access. 

IV 

Denial  of  Service  (DoS)):  Use  of  an  Air  Force  computer/information 
system/network  was  denied  due  to  an  overwhelming  volume  of 
unauthorized  network  traffic. 

V 

Poor  Security  Practice:  An  Air  Force  computer /information 
system/network  was  incorrectly  configured  or  a  user  did  not  follow 
established  policy. 

VI 

Scan/Probe:  Open  ports  on  an  Air  Force  computer /information 
system/network  device  were  scanned  with  no  DoS  or  mission  impact. 

VII 

Malicious  Logic:  Hostile  code  successfully  infected  an  Air  Force 
computer /information  system/network  device.  Unless  otherwise  directed, 
only  those  computers  that  were  infected  will  be  reported  as  a  Category  VII 
incident. 

represent  attacks  against  a  system  in  a  tree  structure,  with  the  goal  as  the  root  node 
and  different  ways  of  achieving  that  goal  as  leaf  nodes”  [57].  If  we  can  understand  how 
an  attack  reaches  its  goal  then  it  will  be  easier  to  stop  the  achievement  of  such  a  goal. 
A  graphical  example  of  an  attack  tree  for  a  house  burglary  is  depicted  in  Figure  2.5 
from  the  Electricity  Sector  Information  Sharing  and  Analysis  Center  (ESISAC)  [22], 

The  reader  may  note,  in  Figure  2.5,  that  “OR”  gates  are  used  to  show  that  only 
one  of  the  options  is  needed  in  order  to  progress  through  the  attack  path.  If  everyone 
robbing  a  home  is:  first,  entering  the  town,  second,  driving  down  the  street,  third, 
entering  the  yard,  and  finally,  going  through  the  front  door  or  the  side  window,  then 
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r=0  r=0  r=0  r=0  r=0 


Figure  2.5:  Example  Threat  Tree,  House  Burglary  [22] 

we  focus  on  the  door  and  the  window.  Although,  no  one  wants  a  burgler  in  town,  a 
more  immediate  concern  is  whether  or  not  the  burgler  can  gain  access  to  the  home. 

Using  an  attack  tree  methodology,  instead  of  only  developing  a  new  virus  sig¬ 
nature  or  finding  a  new  way  to  block  each  individual  new  threat,  we  can  identify 
the  points  of  ingress  in  order  to  protect  them  better.  Concentrating  on  points  of 
ingress  helps  us  to  get  more  protection  from  smaller  changes  as  well  as  better  defense 
in  depth.  Attack  trees  allow  us  to  more  easily  explore  options  for  stopping  groups 
(classes  of  attacks)  in  order  to  make  it  more  difficult  for  the  attacker  to  achieve  their 
goal.  For  example,  if  we  can  change  the  “OR”  gate  at  any  of  the  entry  points  to  an 
“AND”  gate  we  can  force  the  attacker  to  spend  more  time,  money,  and/or  resources 
to  achieve  the  goal. 

Furthermore,  if  our  goal  as  defenders  is  not  only  to  slow  down  the  attackers  but 
to  completely  defend  our  computer  resources  then  understanding  and  implementing 
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all  of  these  classification  techniques  together  will  further  our  cause  and  bring  us  closer 
to  our  goal.  For  example,  continuing  with  the  previous  analogy  of  the  burglar,  if  we 
are  able  to  track  the  burglar’s  progress  ( achieved  access )  then  we  can  more  readily 
focus  our  efforts  on  what  is  needed  to  prevent  further  access.  If  we  know  the  level  of 
automation  that  is  being  used  by  the  burglar  then  we  may  be  able  to  “out  think”  him 
by  doing  something  that  stops  automation  (perhaps  encryption).  If  we  are  able  to 
enumerate  the  attack  types  and  separate  them  into  those  that  can  hurt  us  (unknown 
defense)  and  those  that  don’t  (known  defense),  then  we  can  focus  our  efforts  on  the 
unknown.  Although  as  quoted  by  Sun-Tzu,  a  Chinese  general  and  military  strategist 
in  approximately  400  BC,  “Keep  your  friends  close,  and  your  enemies  closer”,  we 
should  not  forget  about  the  attacks  for  which  we  have  known  defenses.  By  apply¬ 
ing  classification  systems  we  can  better  understand  where  to  protect  our  computer 
systems. 

2.4  Trusted  Computing 

In  an  article  published  by  the  Electronic  Frontier  Foundation  in  October  of 
2003,  Seth  Schoen  reports  “trusted  computing  initiatives  propose  to  solve  some  of 
today’s  security  problems  through  hardware  changes  to  the  personal  computer”  [58]. 
According  to  Schoen,  trusted  computing  architectures  are  being  created  via  Microsoft 
Palladium  (Next  Generation  Secure  Computing  Base),  the  Trusted  Computing  Group 
(TCG),  Intel  Lagrande,  and  AMD  Secure  Execution  Mode  (SEM).  Some  of  the  ideas 
within  this  new  architectural  design  include  memory  curtaining,  secure  input  and 
output,  sealed  storage,  and  remote  attestation.  Memory  curtaining  is  using  hardware 
to  stop  programs  from  reading  or  writing  to  any  memory  space  other  than  their  own. 
Secure  I/O  provides  a  “secure  channel”  from  the  I/O  device  to  the  application  using 
the  device  such  that  the  application  will  be  able  to  trust  its  input  rather  than  worrying 
about  whether  or  not  it  has  been  altered  by  malicious  software  such  as  rootkits.  Sealed 
storage  uses  hardware  to  generate  machine  specific  keys  such  that  data  that  resides 
on  your  machine  can  only  be  decrypted  by  your  machine  and  the  proper  application. 
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Remote  attestation  is  creating  and  sharing  a  cryptographic  hash  of  your  operating 
system  and  any  software  running  on  it  such  that  if  any  changes  are  made  the  remote 
computers  (computers  with  which  you  are  sharing  information)  will  be  able  to  see 
that  changes  have  occurred  and  not  send  information  until  the  situation  has  been 
rectified  [58]. 

The  first  product,  which  originated  from  IBM,  of  the  Trusted  Computing  Group 
is  the  Trusted  Platform  Module  (TPM).  The  TPM  is  a  hardware  component  that 
holds  “keys  used  to  encrypt  data  for  storage  or  transmission”  [38].  By  keeping  the 
keys  and  processing  of  such  in  a  stand  alone  unit,  the  ability  to  sniff  or  otherwise 
guess  the  decryption  keys  is  dramatically  reduced.  Research  is  constantly  underway 
in  the  trusted  computing  area.  Currently,  the  trusted  computing  group  recommends 
the  following  five  steps  [38]  for  implementation  of  trusted  computing: 

1.  Authentication  -  “the  binding  of  an  identity  to  a  subject”  [5] 

2.  Data  protection  -  prevention  of  unauthorized  modification  of  data,  possibly 
through  encryption  or  other  means 

3.  Network  attestation  and  platform  measurement  -  the  implementation  of  authen¬ 
tication  and  data  protection  accross  a  network.  For  example,  authenticating  the 
permissions  of  the  computer  to  be  granted  access  as  well  as  the  patch  levels, 
firewall  setting  [38]. 

4.  Application  protection  -  protection  of  applications  from  malicious  activity  by 
such  things  as  partitioning  memory  so  the  application  can  not  run  out  of  it’s 
own  space 

5.  Content  protection  -  digital  rights  management  to  grant  usage  of  software  only 
to  authorized  persons  [38] 

Efforts  such  as  these  will  help  improve  our  computer  system  defenses  against 
rootkits  by  addressing  some  of  the  behaviors  of  rootkits.  For  example,  rootkits  rou¬ 
tinely  access  information  “outside”  of  their  own  space  such  as  causing  an  application 
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to  jump  to  rootkit  code  which  is  addressed  by  application  protection.  Data  protec¬ 
tion  addresses  an  entire  class  of  rootkits,  that  of  patching.  Network  attestation  will 
also  give  some  protection  against  rootkits  by  helping  verify  those  connecting  to  our 
computers. 

2.5  Exploits  vs  Patches 

As  can  be  seen  in  Figure  2.6,  the  path  from  vulnerabilities  being  discovered,  to 
advisories  being  released  (which  are  followed  by  patches),  is  not  long  but  it  is  long 
enough  to  leave  time  for  the  attacker  to  further  compromise  a  computer  system  or  to 
hide  their  tracks.  In  July  of  2004,  Deborah  Radcliff  of  Security  Focus  reported,  “The 
time  between  vulnerability  discovery  and  exploit,  has  compressed  90  percent  during 
the  past  three  years  the  average  being  11  days  between  discovery  and  exploit  (well 
below  the  23  days  most  enterprises  need  to  patch)...”  [49]. 

Patching  is  a  useful  and  important  step  to  protecting  our  information  systems 
however,  it  is  not  a  panacea.  Scott  Berinato  of  CSO  online  has  an  interesting  view, 
“Slammer  was  unstoppable.  Which  points  to  a  bigger  issue:  Patching  no  longer  works. 
Partly,  it’s  a  volume  problem.  There  are  simply  too  many  vulnerabilities  requiring  too 
many  combinations  of  patches  coming  too  fast”  [4].  The  number  of  vulnerabilities 
continues  to  increase  each  year.  In  2005,  according  to  cert.org  [14],  there  were  5,990 
vulnerabilities  reported,  which  is  more  than  5  times  as  many  as  the  year  2000,  and 
over  35  times  more  than  1995.  While  in  the  first  quarter  of  2006  alone,  there  were  1597 
vulnerabilities  reported  [14],  These  figures  do  not  account  for  the  probable  myriad 
of  vulnerabilities  found  that  have  not  been  reported  which  make  the  total  number  of 
vulnerabilities  that  exist,  increase. 

The  cost  of  patching  can  be  estimated  using  the  following  formula  and  example 
as  given  by  Pete  Lindstrom  in  Information  Security  Magazine,  “(Hours  x  Rate  x 
Systems)  +  (Patch  Failure  x  (Hours  x  Rate  x  Systems))  =  Cost  to  Patch”  [39]. 
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As  an  example  Mr.  Lindstrom  offers  the  following:  “if  it  takes  an  army  of 
$70/hour  technicians  one  hour  to  patch  a  system,  and  there  are  2,000  systems,  the 
cost  is  $140,000.  If  you  estimate  that  5  percent  of  the  patches  fail,  and  figure  an 
average  of  two  hours  of  recovery  time  (which  includes  help  desk  and  IT  support 
activities),  that’s  100  systems  at  $140  each  -  another  $14,000”  [39].  This  comes  to  a 
total  of  $154,000  for  a  single  patch  for  one  company  that  only  has  2000  systems.  This 
can  be  reduced  by  using  automated  patching  but  shows  that  patching  is  expensive.  If 
the  amount  of  patching  can  be  reduced  then  the  total  cost  of  defense  is  thus  reduced 
as  well. 
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Figure  2.6:  Exploitation  Cycle  [44] 


2. 6  Rootkits 

Rootkits,  as  defined  by  Chuvakin,  are  programs  used  to  set  up  and  maintain  an 
environment  [15].  We  add  to  this  definition  to  include,  protection  and  obfuscation  of 
attack  tools  which  exist  on  a  compromised  computer.  According  to  Chuvakin,  there 
are  three  main  types  of  rootkits:  Binary,  kernel  and  library  kits  [15]. 
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However,  there  are  many  ways  to  classify  rootkits  such  as  persistant  vs  memory- 
based.  Persistant  rootkits  are  rootkits  that  survive  on  the  system  (running)  past  a 
reboot  whereas  memory-based  only  rootkits  live  in  memory  and  thus  are  effectively 
destroyed  if  a  reboot  occurs  [11].  Joanna  Rutkowska  in  her  article,  Introducing  Stealth 
Malware  Taxonomy,  divides  the  classification  into  four  types  ranging  from  type  0  to 
type  3  [54], 

1.  Type  0  is  described  as  any  malware  that  uses  only  documented  programming 
methods  and  does  not  directly  interact  with  the  operating  system,  kernel,  or 
other  any  other  processes. 

2.  Type  1  malware  “modifies  those  resources  which  were  designed  to  be  constant, 
like  e.g.  in-memory  code  sections  of  the  running  kernel  and/or  processes.” 

3.  Type  2  malware  then  is  the  counterpart  to  type  1  which  expects  that  malware 
will  modify  resources  that  are  not  constant  such  as  data  sections  (changing 
pointers  in  a  kernel  data  structure  is  cited  as  an  example  of  type  2). 

4.  Type  3  malware  uses  hardware  virtualization  and  exists  outside  of  the  system 
structures.  Rutkowska  also  created  an  example  of  type  3  as  a  proof  of  concept 
called  Blue  Pill. 

In  the  following  subsections  we  will  briefly  explain  some  rootkit  types  in  order 
to  give  a  background  on  rootkits.  This  classification  system  by  type  is  very  use¬ 
ful,  however,  for  this  thesis  and  in  subsequent  chapters  we  will  divide  mainly  into 
three  categories  inspired  by  Hoglund  et  al  [8]:  Hooking,  Patching,  and  Data  Struc¬ 
ture  Manipulation  (DSM).  Each  of  these  categories  can  occur  at  two  levels,  user  and 
kernel  [8]. 

2.6.1  Binary  Rootkits.  Binary  rootkits  replace  common  binary  hies  with 
modified  binary  hies  so  that  when  a  call  to  that  binary  occurs  it  does  as  the  attacker 
wishes  rather  than  as  originally  designed  [15].  An  example  of  such,  is  the  Linux 
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rootkit  TOrn  which  “replaces  login ,  ifconfig,  ps,  du ,  Is,  netstat ,  in. fingered,  find  and 
top  by  manipulated  versions”  [7]. 

2.6.2  Kernel  Rootkits.  Kernel  rootkits  are  probably  the  most  dangerous 
kind  of  rootkit  in  that  they  have  complete  control  over  the  machine  all  the  way  to 
the  root  level.  Kernel  rootkits  use  kernel  calls  to  gain  access  to  hardware  rather 
than  simply  system  calls  which  then  use  kernel  calls.  These  modified  kernel  calls 
then  allow  the  attacker  to  control  what  the  computer  reports  to  its  other  “trusted” 
applications  and  resources.  One  example  of  how  a  rootkit  installs  is  via  a  Loadable 
Kernel  Module  (LKM).  An  LKM  is  used  in  normal  functionality  of  a  system  to  install 
various  applications  and  their  associated  drivers  and  libraries.  However,  rootkits  also 
take  advantage  of  this  functionality  to  insert  themselves  directly  into  the  kernel.  A 
computer  has  various  levels  of  “trust”  which  are  referred  to  as  privileges. 

“A  privilege  in  a  computer  system  is  a  permission  to  perform  an  action.  Ex¬ 
amples  of  various  privileges  include  the  ability  to  create  a  file  in  a  directory,  or  to 
read  or  delete  a  hie,  access  a  device,  or  have  read  or  write  permission  to  a  socket  for 
communicating  over  the  Internet”  [66] .  When  working  with  the  Intel  x86  architecture 
we  refer  to  these  privileges  as  rings.  As  seen  in  Figure  2.7,  Ring  “zero”  is  where  the 
kernel  operates  and  has  full  and  unrestricted  control.  Ring  “three”  is  where  the  user 
has  control  but  must  use  system  calls  in  order  to  request  information  from  hardware 
at  ring  zero.  Ring  “three”  is  also  referred  to  as  “userland”  [8]. 

Of  note  is  that  most  OSs  only  use  ring  zero  and  ring  three  thus  separating  users 
into  somewhat  restricted  or  not  restricted  at  all,  when  it  could  be  very  useful  and 
important  to  have  multiple  levels  of  permission.  For  example,  in  our  banks  we  don’t 
have  only  a  single  differentiation  between  access  modes  (access  to  all  or  access  to 
some).  Rather,  we  have  access  to  some  (e.g.,  user),  access  to  more  than  some  (e.g., 
teller),  access  to  most  (e.g.,  manager),  access  to  all  (e.g.,  owner  with  administrator 
and  manager).  If  we  used  all  four  levels  or  rings  it  would  add  more  levels  of  difficulty 
for  attackers  thus  causing  them  again  to  use  more  resources  in  order  to  succeed.  If 
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Figure  2.7:  Protection  Rings,  Intel  Developer  Manual  [16,36] 


we  can  make  the  amount  of  effort  needed  to  compromise  systems  exceed  that  of  the 
perceived  reward  then  systems  will  be  more  secure  because  fewer  attackers  will  attack. 

One  of  the  advantages  or  problems  (depending  on  your  viewpoint)  with  kernel 
rootkits  is  that  they  can  cause  the  system  to  report  what  the  attacker  wishes,  whether 
it  is  accurate  or  not  [15].  This  includes  examples  such  as:  incorrectly  showing  number 
and/or  names  of  processes  running,  number  and/or  names  of  aircraft  flying,  etc. 

As  documented  by  Hoglund  et  al,  “By  using  a  kernel  hook,  your  rootkit  will  be 
on  equal  footing  with  any  detection  software”  [8] .  By  placing  a  rootkit  in  the  kernel  we 
gain  all  of  the  kernel  controls  that  would  be  granted  to  kernel  level  detection  software. 

2.6.3  Library  Rootkits.  Library  rootkits  work  like  kernel  rootkits  although 
they  run  from  the  user  level  rather  than  the  kernel  level.  By  modifying  libraries 
used  by  kernel  calls  and  system  calls,  the  attacker  can  redirect  or  modify  original 
functionality  to  get  the  desired  effects  of  hie  hiding  or  process  hiding  [15]. 
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2. 7  Summary 

Classification  systems  for  vulnerabilities/attacks  and  fingerprinting  exist  and  are 
helpful  in  the  defense  and  fixing  of  security  vulnerabilities.  However,  classification  only 
by  achieved  access,  automation,  or  attack  type  leaves  a  defender  wondering  how  to  fix 
a  security  flaw.  Certainly,  these  classification  techniques  give  us  further  information 
on  what  exploit  was  used,  but  without  an  attack  tree,  it  is  difficult  to  pinpoint  where 
and  how  to  fix  a  flaw. 

As  discussed  earlier,  the  time  between  discovery  of  a  vulnerability  and  patching 
a  vulnerability  may  be  long  enough  to  install  a  rootkit.  Once  a  rootkit  is  installed 
it  is  much  more  difficult  to  remove  the  compromised  control  of  the  system.  Once 
the  rootkit  has  been  installed,  the  patch  may  fix  the  original  vulnerability  which 
allowed  access  to  the  machine  but  with  a  rootkit  now  present  on  the  machine  it 
does  not  necessarily  need  the  original  vulnerability  to  regain  access.  The  rootkit 
could  be  hiding  any  number  of  other  attack  tools  that  leave  backdoors  and  other 
communication  channels  open,  yet  unseen. 

This  thesis  will  investigate  rootkit  hiding  techniques  (RHTs)  and  some  publicly 
available  rootkit  detection  techniques  (RDTs).  We  then  create  an  attack  tree  from 
which  we  can  identify  deficiencies  in  current  detection  techniques.  The  identified 
deficiencies  can  then  be  used  by  future  researchers  to  increase  the  state  of  the  art  in 
computer  system  defense. 
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III.  Rootkit  Hiding  and  Finding  Techniques 


3.1  Chapter  Overview 

The  main  goal  of  a  rootkit  is  to  create  an  environment  wherein  the  attacker 
may  move  about  freely  and  stealthily.  This  chapter  will  explain  some  of  the  common 
stealthing  techniques  and  give  some  examples  of  each  as  well  as  some  of  the  current 
detection  techniques.  This  thesis  focuses  on  the  stealth  techniques  and  detection 
techniques  although  it  should  be  noted  that  prevention  should  also  be  considered  when 
protecting  your  computer  system.  Prevention  techniques  will  become  very  platform 
and  use  specific.  For  example,  in  UNIX-like  systems,  the  protection  of  /dev/kmem 
and  deactivation  of  kernel  module  support  will  “keep  most  rootkits  outside  of  the 
kernel”  [7]. 

3.2  Rootkit  Stealth 

In  order  to  be  stealthy,  a  rootkit  can  and  should  hide  many  things,  some  of 
which  are  processes  and  hies.  Three  main  hiding  techniques  are  hooking ,  patching , 
and  data  structure  manipulation.  In  general,  hooking  is  changing  the  execution  path 
of  a  call,  patching  is  overwriting  information  in  an  application,  and  data  structure 
manipulation  is  changing  a  data  structure.  Each  of  these  hiding  techniques  will  be 
discussed  in  further  detail.  Jan  K.  Rutkowski  [56]  refers  to  a  similar  separation  in 
techniques  in  Figure  3.1. 

Another  important  concept  to  understand  when  discussing  stealth  techniques 
is  the  difference  between  “hiding  in  plain  sight”  (steganography)  and  “hiding  out  of 
plain  sight” .  In  this  thesis  we  use  “hiding”  to  mean  moving  or  covering  information 
so  that  it  can  not  be  seen  through  the  desired  means  (hiding  out  of  plain  sight). 
Steganography  is  another  technique  that  can  be  used  to  obscure  what  the  user  is 
seeing  so  that  they  do  not  know  that  a  malicious  process  exists  because  they  do  not 
perfectly  recognize  what  they  are  seeing.  Although  steganography  is  normally  hiding 
information  in  pictures  or  other  hies,  an  example  of  steganography  in  this  scenario 
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Figure  3.1:  Rootkit  Technology  Picture  [56] 

would  be:  changing  the  name  of  a  rootkit  to  msdirectx.sys  (as  is  done  by  the  FU 
rootkit  [27])  so  that  it  looks  like  a  legitimate  driver  and  thus,  is  hiding  in  plain  sight. 

3.2.1  Hooking.  Hooking  is  fairly  old  but  as  quoted  from  Hoglund  “...the 
rootkit-world  hasn’t  actually  moved  away  from  system  hooks.  ...most  systems  don’t 
run  even  the  most  basic  of  rootkit  detection  programs,  so  even  SSDT  ( system  service 
descriptor  table)  hooks  are  still  really  effective”  [30]. 

Hooking  works  by  changing  the  original  execution  path  of  some  application  so 
that  the  information  that  it  receives  has  passed  through  the  rootkit  allowing  the 
rootkit  to  scrub  the  data,  effectively  allowing  the  rootkit  to  hide  itself,  and  anything 
else  it  chooses,  from  view  [8].  An  example,  is  nicely  illustrated  by  Hoglund  et  al, 
in  their  book  “Rootkits:  Subverting  the  Windows  Kernel”  which  is  recreated  in  Fig¬ 
ure  3.2.  If  we  follow  the  diagram  we  see  that  information  can  be  scrubbed  by  the 
rootkit,  as  desired  by  the  attacker,  prior  to  returning  to  the  source  function.  First 
the  function  being  rooted  (source  function)  is  overwritten  with  a  jump  to  the  rootkit 
code  (the  detour),  next  the  rootkit  code  (the  trampoline)  is  executed.  Part  of  the 
rootkit  code  is  to  execute  the  overwritten  data  from  the  source  function  so  that  the 
proper  context  is  in  the  proper  registers  when  the  original  code  is  executed.  This  is 
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then  followed  by  a  jump  to  the  original  function  plus  an  offset  (the  target)  to  account 
for  the  information  that  was  overwritten.  When  the  target  function  is  done  executing 
it  will  return  the  results  to  the  rootkit  (the  detour/new  calling  function)  which  can 
then  scrub  the  data  that  it  desires.  The  rootkit  will  then  return  the  scrubbed  data 
to  the  original  calling  function. 


Figure  3.2:  Temporal  ordering  of  a  detoured  function  [8] 

Kernel  Level  Hooking  -  The  three  most  common  areas  to  hook  in  the  Windows 
kernel,  according  to  Hogland,  are  the  System  Service  Descriptor/Dispatcher  Table 
(SSDT),  Interrupt  Descriptor  Table  (IDT),  and  the  I/O  Request  Packet  (IRP)  Func¬ 
tion  tables  [8]. 

1.  The  SSDT  is  the  table  in  Windows  which  holds  a  list  of  all  system  services  by 
ID  and  address.  As  reported  by  Prasad  Dabak  et  al  in  their  book  Hooking  Windows 
NT  System  Services,  Windows  “system  services  can  be  considered  the  equivalent  of 
system  calls  in  UNIX”  [20].  System  services  and  system  calls  “represent  the  funda¬ 
mental  interface  for  any  user- mode  application  or  subsystem  to  the  kernel”  [20]. 

2.  The  IDT  is  the  table  which  holds  the  interrupt  identifiers  and  their  associ¬ 
ated  addresses.  An  IDT  exists  both  in  Windows  and  UNIX-like  operating  systems. 
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In  Windows,  “The  Interrupt  Descriptor  Table  (IDT)  is  an  array  of  8  byte  interrupt 
descriptors  in  memory  devoted  to  specifying  (at  most)  256  interrupt  service  routines. 
The  first  32  entries  are  reserved  for  processor  exceptions,  and  any  16  of  the  remain¬ 
ing  entries  can  be  used  for  hardware  interrupts.  The  rest  are  available  for  software 
interrupts”  [47]. 

3.  The  I/O  request  packet  is  where  information  needed  to  process  an  I/O 
request  is  stored  within  the  I/O  manager  and  is  used  to  represent  the  operation  as 
it  is  processed  throughout  the  system  [53].  Because  the  IRP  is  a  location  that  stores 
information  about  an  I/O  operation  it  becomes  easy  to  see  how  this  would  be  a 
valuable  place  to  hook  for  a  rootkit  and  a  place  to  protect  as  a  defender. 

3.2. 1.1  Hooking  the  System  Service  Descriptor  Table  (SSDT).  This 
hooking  method  works  on  the  SSDT  and  is  not  constrained  to  a  single  application. 
According  to  James  Butler  and  Sherri  Sparks  in  their  article,  Windows  Rootkits  of 
2005,  the  SSDT  is  where  the  “actual  implementation  of  the  operating  system  functions 
are  contained”  [10].  SSDT  hooking  works  by  replacing  the  addresses  of  ZW*  functions 
with  the  address  of  rootkit  code  ( “ZW  routines  provide  a  set  of  system  entry  points 
that  parallel  some  of  the  executive’s  system  services.  Calling  a  ZW  routine  from 
kernel-mode  code  results  in  a  call  to  the  corresponding  system  service”  [17]).  This 
gives  the  opportunity  for  the  rootkit  code  to  manipulate  information  that  would  have 
been  returned  to  any  calling  application.  ZW*  functions  are  functions  exported  by 
the  kernel  for  usage  by  other  kernel  functions  and  device  drivers.  When  a  ZW* 
function  is  called,  usually  by  an  NT*  function  (system  call)  it  returns  the  address 
corresponding  to  the  NT*  function  which  was  stored  in  the  SSDT  [8].  As  can  be  seen 
in  Figure  3.3,  the  hook  simply  overwrites  the  address  of  the  hooked  kernel  function 
with  the  address  of  the  rootkit  so  that  any  application  calling  the  hooked  function  will 
have  it’s  execution  path  pass  through  the  rootkit  giving  the  rootkit  the  opportunity 
to  scrub  or  alter  data. 
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Normal  Application  Call 


Application  Call  w/SSDT  hook 


Figure  3.3:  Example  SSDT  Hook 


3.2. 1.2  Hooking  the  Interrupt  Descriptor  Table  (IDT).  This  hooking 
method  works  on  interrupts  and  is  not  constrained  to  a  single  application  or  a  single 
operating  system.  IDT  hooks  work  fundamentally  differently  than  SSDT  hooks  in 
that  with  an  SSDT  hook  the  idea  is  to  insert  the  rootkit  into  the  execution  loop  so 
that  it  can  scrub  the  output  of  a  legit  function,  thus  returning  incorrect /incomplete 
data  to  the  calling  application.  The  IDT  hook,  on  the  other  hand,  does  not  insert 
in  the  execution  loop.  Rather,  the  IDT  hook  breaks  the  loop  and  calls  the  rootkit 
code  instead  of  the  originally  intended  code  as  can  be  seen  in  Figure  3.4.  This  allows 
the  rootkit  to  identify  that  there  is  a  call  from  a  particular  application  so  that  it  can 
identify  and  then  allow  or  block.  This  method  works  by  replacing  the  address  of  a 
legitimate  interrupt  in  the  IDT  with  an  entry  to  some  rootkit  code  [8].  This  causes 
the  rootkit  code  to  be  called  every  time  the  interrupt  occurs.  Altering  the  IDT  works 
both  in  Windows  and  in  Linux  simply  by  changing  the  address  of  a  legitimate  jump 
to  the  address  of  rootkit  code  in  the  IDT. 

3.2. 1.3  Hooking  the  I/O  Request  Packet  (IRP)  Function  Tables.  IRP 
hooks,  just  like  IDT  hooks,  are  not  returned  to  so  creating  a  hook  that  calls  to  rootkit 
code  is  necessary  to  alter  information  coming  from  the  intended  driver.  An  illustration 
of  an  IRP  hook  can  be  seen  in  Figure  3.5. 
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Normal  IDT  Usage  IDT  Usage  w/IDT  hook 


Figure  3.4:  Example  IDT  Hook 


Figure  3.5:  Example  IRP  Hook  [9] 


Figure  3.6,  shows  a  normal  device  driver  physical  structure.  Once  hooked  the 
diagram  would  change  the  “Device  Drivers”  box  to  “rootkit  driver”.  It  is  necessary 
to  write  a  completion  function  so  that  the  information  from  the  original  device  driver 
can  be  sanitized  [8]. 

A  common  theme  among  the  Kernel  Mode  hooks  is  that  they  each  overwrite  an 
address  in  a  kernel  data  structure  with  a  rootkit  address  and  they  are  not  application 
specific. 

User  Level  Hooking  can  also  be  done  from  the  user  level  or  “Userland”  using 
API  hooking.  API  hooking  is  overwriting  or  modifying  sections  of  hies/processes  used 
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Figure  3.6:  Device  Driver  Physical  Structure,  Microsoft  [18] 


by  an  application  to  do  other  than  intended  purposes  or  to  report  other  than  accurate 
information. 

There  are  two  common  kinds  of  userland  API  hooking  processes;  Import  Address 
Table  (IAT)  hooking,  and  Inline  function  hooking.  The  IAT  is  a  data  structure 
belongs  to  and  resides  within  the  address  space  of  most  application’s.  The  IAT  holds 
the  imported  addresses  of  functions  to  which  the  application  plans  to  jump.  Inline 


function  hooking  is  simply  placing  “extra”  instructions  into  a  function.  The  resultant 
execution  path  of  inline  function  hooking  can  be  seen  in  Figure  3.2. 


3. 2. 1-4  Import  Address  Table  (IAT)  Hooking.  IAT  hooking  works  like 
SSDT  hooking  except  it  operates  on  the  IAT  table  rather  than  the  SSDT  table.  IAT 
is  also  a  hook  per  application  rather  than  a  hook  for  all  applications.  The  SSDT 
holds  the  addresses  of  system  services  that  applications  use,  while  each  application 
has  its  own  IAT.  An  example  of  IAT  hooking  can  be  seen  in  Figure  3.7.  Normal  flow 
would  be:  Application,  IAT,  called  function.  This  can  occur  because  the  IAT  can  be 
overwritten  to  jump  to  the  rootkit  instead  of  the  intended  function.  This  allows  the 
rootkit  to  be  the  caller  of  the  intended  function  which  gives  the  ability  to  clean/alter 
the  returned  information  [8]. 


Figure  3.7:  Normal  execution  path  vs.  hooked  execution  path 
for  an  IAT  hook  [9] 


3.2. 1.5  Inline  function  hooking.  Inline  function  hooking  is  basically 
the  same  thing  as  IAT  hooking  except  the  control  of  the  target  function  is  taken  by 
replacing  the  first  5  bytes  of  the  target  function  with  a  jump  to  the  rootkit  function. 
The  rootkit  function  then  calls  the  target  function  using  the  previously  saved  first 
5  bytes.  In  this  way  the  target  function  will  do  its  intended  operation  but  then 
return  the  results  to  the  rootkit  rather  than  the  calling  application.  The  rootkit 
will  then  modify  the  data  as  needed  and  return  the  erroneous  results  to  the  calling 
application  [8].  Figure  3.2,  shows  how  the  inline  hooking  would  work.  The  actual 
usage  of  inline  function  hooking  would  look  like  Table  3.1,  which  shows  the  normal 
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preamble  of  a  Windows  function  and  the  modified  preamble.  Inline  function  hooking 
is  very  similar  to  the  following  rootkit  concept  which  is  patching. 


Table  3.1:  Inline  Function  Hook  [10] 


Original  Preamble 

Code  Bytes 

Assembly 

8bff 

mov  edi,  edi 

55 

push  ebp 

8bec 

mov  ebp,  esp 

Modified  Preamble 

Code  Bytes 

Assembly 

e9  xx  xx  xx  xx 

jmp  xxxxxxxx 

3.2.2  Patching.  Patching  is  overwriting  a  binary  such  that  it  performs  in  a 
different  way  than  it  originally  performed.  An  example  of  malicious  patching  would  be 
to  analyze  a  program  to  find  where  the  conditionals  are  located.  One  such  conditional 
could  be  the  Jump  if  Not  Zero  (JNZ)  instruction  which  can  be  used  to  stop  access 
to  a  program  if  the  key  entered  is  not  correct/valid.  This  can  easily  be  patched 
with  the  use  of  a  hexadecimal  editor.  We  simply  search  for  the  appropriately  located 
hexidecimal  value  75  (the  JNZ  assembly  instruction)  and  change  it  to  hexidecimal 
value  EB  (the  JMP  assembly  instruction).  This  change  effectively  says  “jump  to  the 
next  instruction  even  if  the  right  key  is  not  entered”  thus  making  the  “nag  screen”  no 
longer  appear.  Patching  is  also  used  to  improve  or  fix  otherwise  incomplete/incorrect 
code. 


3.2.3  Data  Structure  Manipulation.  Data  Structure  Manipulation  (DSM)  is 
modifying  a  data  structure  such  that  it  no  longer  works  in  the  same  way.  An  example 
of  DSM  is  Direct  Kernel  Object  Manipulation  (DKOM).  Other  RHTs  remove  data  or 
change  data  while  DSM  is  changing  the  structure  such  that  the  data  still  exists  it  is 
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just  not  seen  or  used  in  the  same  fashion.  An  excellent  example  of  this  technique  is 
DKOM  as  implemented  by  FU  which  is  described  in  Section  3. 2. 3.1, 


3.2.3. 1  Direct  Kernel  Object  Manipulation  (DKOM).  One  of  the 
objects  within  the  Windows  kernel  that  can  be  modified  is  the  executive  process 
block  (EPROCESS)  which  contains  information  about  its  associated  process  as  well 
as  pointers  to  other  needed  data  structures  [53].  In  this  scenario  used  by  FU  and 
FUTo  rootkits,  DKOM  modifies  the  doubly  linked  list  of  processes  such  that  the 
pointers  in  the  list,  point  around  the  process  to  be  hidden,  effectively  removing  it 
from  the  list.  Once  the  process  has  been  removed  from  the  list  it  can  still  continue 
to  execute  because  the  execution  occurs  through  the  associated  threads  rather  than 
from  one  process  to  the  next.  This  does,  however,  effectively  remove  the  process  from 
any  queries  that  attempt  to  walk  this  linked  list  of  processes. 


Figure  3.8:  Direct  Kernel  Object  Manipulation  [8] 


The  removal  of  the  EPROCESS  block  from  the  list  occurs  by  making  the  forward 
link  (FLINK)  and  backward  link  (BLINK)  pointers,  in  the  EPROCESS  block  of  the 
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process  to  be  hidden,  point  to  each  other,  while  making  the  neighboring  blocks  point 
to  each  other.  The  EPROCESS  block  is  accessed  by  using  the  kernel  processor  control 
block  to  view  the  current  thread  which  points  to  the  ETHREAD  block,  which  then 
points  to  the  EPROCESS  block.  Each  EPROCESS  block  is  connected  via  a  doubly 
linked  list  to  other  EPROCESS  blocks.  Figure  3.8,  shows  the  modification  of  the 
FLINK  and  BLINK  pointers  [16]. 

3. 2. 3. 2  Future  DSM  Targets.  Data  structure  manipulation  is  one  of 
the  cutting  edge  rootkit  techniques  and  as  such  will  continue  to  develop.  One  obvious 
location  for  implementation  of  such  techniques  will  be  in  the  scheduler.  Currently 
there  is  a  proof  of  concept  (used  for  detection)  called  Klister,  developed  by  Joanna 
Rutkowska  in  2003.  Klister  allows  the  scheduler/dispatcher  lists  to  be  read,  showing 
the  currently  running  threads  which  are  used  to  schedule  time  on  the  processor  [12,13]. 
Threads  can  then  be  checked  to  see  which  process  they  belong  to,  thus  showing  an 
accurate  view  of  which  processes  are  running  regardless  of  hiding  techniques  to  date 
such  as  DKOM.  This  RDT  could  be  overcome  at  least  in  theory  by  manipulating  the 
scheduler/dispatcher  such  that  the  desired  thread(s)  are  hidden. 

3.2.4  Virtual  Machine  and  Virtual  Memory.  Virtual  machine  and  virtual 
memory  rootkits  are  some  of  the  most  cutting  edge  techniques.  Virtual  machine 
rootkits,  such  as  Blue  Pill  by  Joanna  Rutkowska  [54,55],  seek  to  turn  the  host  OS 
into  a  virtual  machine  by  lifting  it  and  inserting  the  rootkit  below  as  the  host  OS.  In 
this  way  the  rootkit  would  have  access  to  anything  and  everything  in  the  now  virtual 
OS  but  the  virtual  OS  would  have  no  indication  of  the  rootkit’s  presence.  Virtual 
memory  rootkits  seek  to  monitor  their  own  memory  section  address  space  such  that 
when  another  process  tries  to  read  it  they  are  redirected.  An  example  of  virtual 
memory  rootkits  is  cited  and  explained  in  further  detail  in  Section  3.2.9. 
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3.2.5  Hardware.  Although  inserting  rootkits  at  the  hardware  level  may  be 
feasible,  as  documented  by  John  Heasman  in  his  paper  Implementing  and  Detecting 
a  PCI  Rootkit  [28],  for  this  thesis,  we  will  not  delve  into  as  much  detail  on  the  issue. 

Summary  of  Stealth  Techniques  -  As  discussed  earlier  the  three  categories  used 
by  rootkits  to  hide  are  Hooking  (changing  the  execution  path),  Patching  (overwriting 
an  executable),  and  Data  Structure  Manipulation  (changing  the  data/structure  of  an 
object).  These  same  techniques  are  used  to  attack  both  Windows  and  Linux  operating 
systems.  In  Figure  3.9  we  can  see  how  the  three  categories  fit.  All  software  exists 
as  data  (binary  code)  which  can  potentially  be  overwritten  or  patched  (inner-most 
circle).  This  data  is  generally  in  some  format,  object  or  construct  of  some  sort  (list, 
array,  etc)  which  is  what  is  hooked,  in  order  to  change  the  execution  path  (middle 
circle).  These  constructs  all  interact  in  one  form  or  another  with  their  environment 
such  as  the  OS  which  is  their  data  structure.  This  data  structure  can  potentially  be 
manipulated  (outer  circle).  For  example,  changing  or  removing  pointers  to  an  object. 
It  is  important  as  a  defender  and  an  attacker  to  understand  where  all  the  tables  with 
important  data  exist,  and  where  control  exists.  Hardware  certainly  puts  a  slightly 
different  spin  on  these  categories  but  still  roughly  maintains  these  categories  simply 
by  containing  these  categories  within  a  new  set  of  hardware.  For  example,  if  a  PCI 
card  is  added  to  a  machine  in  order  to  protect  the  machine,  the  PCI  card  itself  would 
have  these  three  categories  within  itself.  The  three  categories  would  also  then  exist 
on  the  same  machine  but  outside  of  the  PCI  card  thus  giving  two  distinct  locations 
to  check. 

The  following  sections  will  give  some  examples  of  the  three  categories  of  rootkits. 

3.2.6  Rootkit  Examples  -  Hooking.  As  previously  described,  hooking  works 
by  changing  the  original  execution  path  of  some  application  so  that  the  information 
that  it  receives  has  passed  through  the  rootkit  allowing  the  rootkit  to  scrub  the  data, 
effectively  allowing  the  rootkit  to  hide  itself  and  anything  else  it  chooses  from  view  [8]. 
The  following  are  some  examples  of  hooking  rootkits: 
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3.2.6. 1  AFX  -  Windows  RK.  Hides  by  hooking  the  SSDT  as  described 
previously.  AFX,  as  will  be  shown  later,  can  be  detected  by  other  rootkits  at  the  same 
level  or  by  checking  other  areas  of  the  operating  system  for  existence  of  AFX  [62], 

3. 2. 6. 2  Vanquish  -  Windows  RK.  Works  by  hooking  of  API  calls  as 
well  as  patching.  Once  the  API  is  hooked  the  function  is  patched  so  that  the  hook  can 
subsequently  be  “undone”  [69].  The  hook  no  longer  needs  to  exist  once  the  function 
is  patched  because  the  function  will  carry  out  the  rootkit  functionality  due  to  the 
patch. 


3. 2. 6. 3  Hacker  Defender  -  Windows  RK.  The  readme  hie  for  hacker 
defender  describes  hacker  defender  as  follows:  “...  rewrite  few  memory  segments  in 
all  running  processes.  ...able  to  hide  hies,  processes,  system  services,  system  drivers, 
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registry  keys  and  values,  open  ports,  cheat  with  free  disk  space.  ...also  masks  its 
changes  in  memory  and  hides  handles  of  hidden  processes.  . .  .installs  hidden  backdoors, 
register  as  hidden  system  service  and  installs  hidden  system  driver”  [32],  This  rootkit 
works  by  hooking  various  API  functions  to  hook  ALL  processes  in  the  system  [32], 

3. 2. 6. 4  Adore  BSD  0.34  -  BSD  RK.  Adore  is  a  kernel  loaded  module 
that  is  used  to  hide  hies,  processes  and  network  connections.  It  uses  a  second  module 
to  delete  itself  from  the  kernel  data  structures.  The  SSDT  method  described  earlier 
is  used  to  insert  the  rootkit  via  the  overwriting  of  15  system  calls  [7].  Subsequent 
versions  of  Adore  such  as  Adore-NG  1.41  use  the  same  kernel  module  loading  but 
then  infect  other  areas  of  the  kernel  such  as  the  virtual  filesystem  layer  such  that  it 
does  not  have  to  hide  itself  because  it  becomes  part  of  other  modules  [7] . 

3.2.7  Rootkit  Examples  -  Patching.  Patching  is  overwriting  a  binary  such 
that  it  performs  in  a  different  way  than  it  originally  performed.  The  following  are 
some  examples  of  patching  rootkits: 

3.2.7. 1  eEye  Bootrootkit  -  Windows  RKs.  Created  by  Derek  Soeder 
and  Ryan  Permeh  of  eEye  Digital  Security,  this  proof  of  concept  proves  use  of  the 
boot  sector  to  create  a  rootkit.  It  uses  a  hook  into  the  interrupt  13h  in  order  to  patch 
the  OS  loader  and  then  hooks  ndis.sys  [61]  which  is  the  Windows  Network  Driver 
Interface  Specification.  NDIS.sys,  according  to  related  forums,  holds  a  “collection  of 
routines  that  applications  can  invoke  to  perform  network-related  operations”  [50]. 

3. 2.7. 2  SucKIT  1.3b  -  Linux  RK.  SucKIT  is  a  Linux  rootkit  designed 
by  Silvio  Cesare  [15]  and  includes  mechanisms  for  reboots  and  a  backdoor.  This 
rootkit  is  listed  in  the  patching  section  because  majority  of  its  techniques  rely  on 
patching  however,  as  you  will  see  many  of  the  ideas  from  other  techniques  are  used. 
This  rootkit  installs  itself  by  doing  a  search  on  the  memory  of  the  system  to  find 
the  location  of  kmalloc()  (the  function  used  to  allocate  memory  in  the  kernel)  and 
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the  SSDT.  Once  found,  the  address  of  kmalloc()  is  placed  in  an  unused  entry  of  the 
SSDT.  The  entry  that  now  exists  in  the  SSDT  is  called  and  used  to  allocate  kernel 
memory,  and  the  rootkit  is  subsequently  loaded  into  that  memory  space.  The  SSDT 
entry  used  is  then  overwritten  to  jump  to  the  rootkit  space.  The  /sbin/init  hie  is  also 
overwritten  or  modified  in  order  to  reload  this  rootkit  after  a  reboot.  However,  this 
rootkit  also  copies  the  original  hie  before  overwriting  so  that  when  a  query  is  made 
to  the  hie,  the  rootkit  can  redirect  the  query  to  the  original  but  renamed  hie.  This 
saving  and  redirecting  stops  detection  via  a  simple  checksum.  While  using  patching  as 
a  means  for  loading,  this  rootkit  also  takes  advantage  of  the  SSDT  to  hook  24  system 
calls.  However,  the  implementation  of  the  hook  is  slightly  different.  The  SSDT  is 
actually  copied  and  then  modified.  The  IDT  is  then  hooked  to  have  subsequent 
system  calls  jump  to  the  modified  SSDT.  This  stops  detection  of  the  rootkit  by  SSDT 
inspection  [7]. 

3. 2. 7. 3  TOrn  8  -  Linux  RK.  TOrn  hides  process  information  by  patch¬ 
ing  system  libraries  such  as  libproc.a  in  linux  systems.  Libproc.a  is  “used  for  relaying 
the  process  information  from  the  kernelspace  (via  /proc  hie  system)  to  user  space 
utilities  such  as  /bin/ps  and  top ”  [15]. 

3.2.8  Rootkit  Examples  -  Data  Structure  Manipulation. 
data  structure  such  that  it  no  longer  works  in  the  same  way 
examples  of  DSM  rootkits: 

3.2.8. 1  FU  and  FUTo  -  Windows  RKs.  FU  and  FUTo  hide  by  using 
DKOM  to  redirect  pointers  within  the  EPROCESS  block  in  Windows  OSs  as  de¬ 
scribed  in  Section  3.2.3. 1.  A  possible  way  to  subvert  this  hiding  technique  is  to  follow 
each  thread  from  the  Kernel  Process  Control  Block  (KPCB)  to  its  Ethread  and  sub¬ 
sequently  to  its  EProcess,  thus  finding  each  process  by  iterating  through  threads  in 
the  KPCB  rather  than  processes  in  the  EProcess  blocks.  A  similar  concept  is  shown 
in  “Klister”  as  previously  mentioned. 


DSM  modifies  a 
The  following  are 
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3.2.9  Rootkit  Examples  -  Virtual  Memory. 


3.2.9. 1  Shadow  Walker  -  Windows  RKs.  Shadow  Walker  uses  virtual 
memory  to  hide  itself  and  its  malicious  processes  from  detection.  As  reported  by 
James  Butler  and  Sherri  Sparks,  Shadow  Walker  is  a  proof  of  concept  that  covers 
three  concerns  in  virtual  memory  [11],  First,  it  detects  when  something  other  than 
itself  is  attempting  to  read  its  memory  space.  This  is  done  by  identifying  the  dif¬ 
ference  between  read,  write,  and  execute.  Differentiating  between  read,  write,  and 
execute  (read/execute,  write/execute)  is  accomplished  by  marking  the  page  table  en¬ 
tries  (PTEs)  as  “non  present”  and  hooking  the  page  fault  handler.  This  allows  Shadow 
Walker  to  monitor  access  to  these  pages.  Once  it  has  been  clearly  identified  which 
of  these  is  being  used,  the  RK  must  be  able  to  fake  the  read  or  send  back  erroneous 
information  to  would-be  detection  utilities.  If  the  access  is  a  read  access,  Shadow 
Walker  returns  erroneous  information  to  the  application,  otherwise  Shadow  Walker 
runs  itself  as  intended.  The  last  thing  addressed  by  Shadow  Walker  is  that  there  is 
little  identifiable  performance  degradation  because  the  increase  in  the  number  of  page 
faults  generated  is  minimal.  [11] 

Other  UNIX  based  rootkits  include:  (usermode)-lkr,  ark,  (kernelmode)-Knark(written 
by  Creed). 

As  can  be  seen  in  Table  3.2,  there  are  many  rootkits.  Each  rootkit  studied  fits 
into  at  least  one  of  the  three  categories,  namely  hooking,  patching,  or  DSM.  However, 
as  can  also  be  seen  in  the  table  there  are  at  least  two  rootkits  which  have  treaded 
on  new  territory,  that  of  VM  and  Hardware.  Both  of  these  new  territories  still  fall 
within  hooking,  patching  and  DSM  as  described  earlier  but  because  of  the  newness 
were  left  in  the  table  as  separate  entries. 

3.3  Rootkit  Detection 

There  are  many  techniques  to  detect  rootkits,  however,  as  stated  elegantly  by 
the  writers  of  www.hxdef.org  “For  an  attacker  one  security  hole  is  enough  to  win 
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Table  3.2:  Rootkit  Examples 


Rootkit 

Hooking 

Patching 

DSM 

VM 

Hardware 

AFX 

X 

X 

Vanquish 

X 

X 

Hacker  Defender 

X 

X 

Adore/ava 

X 

X 

SucKIT 

X 

X 

Apropos 

X 

X 

TOrn 

X 

FU 

X 

FUTo 

X 

deep door 

X 

peligroso 

X 

firewalk 

X 

prrf 

X 

phide2 

X 

Shadow  Walker 

X 

X 

Blue  Pill 

X 

X 

eEye  Bootrootkit 

X 

X 

HE4Hook 

X 

NTRootkit 

X 

NTIllusion 

X 

VideoCardKit 

X 

X 

Sony B MG  XCP 

X 

wootkit 

X 

RK 

X 

the  game.  One  can  say  the  role  of  attacker  is  easier.  But  if  you  want  to  fight  the 
attacker  you  can’t  produce  an  antirootkit  solution  that  just  fixes  or  protects  one  weak 
point.  You  have  to  fix  all  those  points.  What’s  more,  you  can’t  have  weak  points 
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in  that  solution.”  [31]  Rootkits  are  constantly  evolving  and  improving  thus  rootkit 
detection  methods  must  get  better  and  more  complete.  There  may  or  may  not  exist  a 
single  rootkit  detector  that  can  find  all  rootkits.  James  Butler  and  Sherri  Sparks,  in 
their  article  Windows  Rootkits  of  2005  [12],  divide  rootkit  detection  techniques  into 
five  categories:  signature  based,  behavioral/heuristic,  crossview,  integrity  based  and 
hardware  detection  [12]. 

1.  Signature  based  detection  is  scanning  data  for  a  pattern  which  comprises  a 
“fingerprint”  that  is  unique  to  a  particular  entity  [11]. 

2.  Behavioral/heuristic  based  detection  seeks  to  identify  actions  or  patterns  that 
are  abnormal  to  system  operation.  These  detection  techniques  “work  by  recognizing 
deviations  in  “normal”  system  patterns  or  behaviors”  [11]. 

3.  Crossview  based  detection  uses  the  idea  of  data  redundancy  (the  existence 
of  equivalent  data  in  more  than  one  location)  to  find  “answers”  from  multiple  sources 
that  should  be  the  same  in  order  to  identify  discrepancies  [11].  An  example  of  such  is 
a  child  asking  her  mother  for  permission  to  go  to  the  park  and  then  asking  her  father 
permission  for  the  same  thing.  The  two  answers  should  be  the  same;  however,  the 
difference  in  answers  is  what  is  exploited  by  the  child. 

4.  Integrity  based  detection  compares  a  known  good  entity  with  a  suspected 
entity  in  order  to  verify  the  correctness/accurateness  of  the  suspect  entity.  An  example 
of  such  is  comparing  “a  current  snapshot  of  the  filesystem  or  memory  with  a  known, 
trusted  baseline”  [11]. 

5.  Hardware  based  detection  uses  a  piece  of  hardware  to  implement  one  or 
more  detection  techniques  such  as  signature,  heuristic,  crossview,  or  integrity  based 
detection  while  separating  itself  from  the  suspected  operating  system  [11].  Hardware 
detection  makes  it  more  difficult  for  the  attacking  entity  to  corrupt  the  results  of  the 
hardware. 

As  important  as  each  of  these  categories  are;  we  have  come  to  a  different  solution 
for  categorization.  If  it  is  possible  to  classify,  such  that  a  taxonomy  is  developed,  then 
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we  will  be  able  to  move  more  quickly  in  our  research  to  detect  “all”  rootkits  because 
“all  rootkits”  would  fit  into  the  taxonomy.  In  an  effort  to  move  toward  this  taxonomy 
we  propose  the  following  categorization.  We  propose  detection  classes,  techniques, 
and  implementations  each  of  which  will  have  its  sub  categories. 

Detection  classes  give  categories  that  show  ways  in  which  it  might  be  possible  to 
detect.  Surely  there  may  be  classes  that  we  have  not  yet  discovered,  but  to  date  we  see 
these  classes  as:  Static  (Analysis/Detection  based  on  the  knowledge  of  characteristics 
possessed  by  an  entity  such  as  a  rootkit)  and  Behavioral  (Analysis/Detection  based 
on  the  behavior  or  lack  of  absence  of  behavior  of  an  entity). 

Detection  techniques  are  the  subsets  of  detection  classes  which  describe  how  the 
detection  classes  may  work.  Subsets  of  the  static  class  are:  signature  (analysis/detec¬ 
tion  of  an  entity  through  identified  patterns  or  sequences)  and  integrity  (analysis/de¬ 
tection  of  an  entity  through  verification  and  comparison  of  known  good  patterns  or 
sequences  with  suspect  patterns  or  sequences).  Subsets  of  the  behavioral  class  are: 
anomaly  (analysis/detection  of  and  entity  through  identification  of  unknown  behav¬ 
ior)  and  signature  (as  previously  defined).  Each  of  these  techniques  can  take  on  an 
aspect  of  time  via  a  snapshot  of  past,  present  or  future  states. 

Detection  implementations  are  the  ways  in  which  we  may  implement  detection 
techniques  or  combinations  of  detection  techniques  as  is  the  case  with  crossview.  The 
identified  implementations  to  date  are:  crossview  (the  comparing  of  two  or  more 
inputs,  whether  from  the  same  technique  or  various,  in  order  to  achieve  detection), 
remote  attestation  (comparing  “outside”  information  with  information  on  a  “suspect” 
machine),  cognitive/human,  hardware,  software,  and  memory  tracking  (detection  of 
anomalies  via  known  memory  behaviors). 

We  arrive  at  this  categorization  by  analyzing  detection  starting  with  Figure  3.10. 
In  this  graph  we  divide  detection  into  two  areas;  “What  it  is”  (What  are  we  trying 
to  detect),  and  “What  it  is  doing”  (What  is  the  entity,  that  we  are  trying  to  detect, 
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doing).  The  first  branch  is  then  followed  further  to  identify  two  ways  of  detecting 
“What  it  is” ;  Is  it  there,  or  can  we  verify  that  it  is  not  there. 


Figure  3.10:  Detection  Classification  1 


“What  it  is”  refers  to  what  is  the  rootkit  which  we  are  trying  to  detect  and  a 
static  view  (snapshot)  of  either  what  it  looked  like  in  the  past,  what  it  currently  looks 
like,  or  what  it  will  look  like  in  the  future.  This  requires  the  knowledge  of  the  rootkit 
and  what  it  looks  like  at  various  stages.  “What  it  is  doing”  refers  to  the  actions  or 
behavior  of  the  rootkit  or  the  actions  and  behaviors  caused  by  the  rootkit  (the  actions 
and  behaviors  of  the  surrounding  system).  Figure  3.11,  shows  the  same  graph  with 
the  new  classes  and  techniques  inserted.  Figure  3.12  adds  detection  implementations 
and  other  known  inputs  to  a  system. 

Other  known  inputs  to  a  system  that  might  help  in  detection  could  be  intel¬ 
ligence  (obtained  from  other  sources,  i.e. ,  someone  said  there  is  a  rootkit  on  my 
machine),  witness  (the  data,  object,  or  structure  affected  by  the  victim),  victim  (the 
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Figure  3.11:  Detection  Classification  2 


data,  object,  or  structure  that  was  manipulated),  attacker  (the  manipulator  of  data, 
object  or  structure),  and  outside  sources  (such  as  a  news  report  or  other  information). 

If  we  incorporate  all  of  our  classes,  techniques,  implementations  and  other  inputs 
into  a  graph  and  connect  them  via  arrows  showing  inputs  we  obtain  a  very  busy 
Figure  3.12  which  shows  how  each  of  the  classes,  techniques,  and  implementations 
interconnect. 

3.3.1  Behavioral  Detection  Class.  Behavioral  based  detection  seeks  to  iden¬ 
tify  actions  or  patterns  that  are  abnormal  to  system  operation.  These  detection  tech¬ 
niques  “work  by  recognizing  deviations  in  normal  system  patterns  or  behaviors”  [11]. 
A  lack  of  pattern  or  action  is  also  a  behavior,  but  it  is  a  behavior  of  the  system  rather 
than  a  behavior  of  the  entity  being  investigated.  For  example,  if  an  application  is 
launched  that  should  create  a  particular  pattern  and  it  does  not  create  said  pattern 
then  that  is  anomalous.  Behavioral  detection  uses  two  main  techniques:  anomaly  and 
signature.  Behavioral  detection  can  also  be  an  input  to  many  detection  implementa¬ 
tions  such  as:  Crossview,  hardware,  software,  cognitive/human,  remote  attestation, 
and  memory  tracking. 
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Figure  3.12:  Detection  Classification  3 


3.3.2  Static  Detection  Class.  The  static  detection  class  is  defined  by  what 
the  target  entity  is  comprised  of  or  its  attributes,  such  as  code.  Static  detection 
currently  has  two  techniques,  signature  and  integrity.  Signature  based  detection  is 
defined  as  a  particular  set  (predefined  set)  of  instructions  or  code  which  are  to  be 
found  and  acted  upon  (i.e.,  identified,  blocked,  removed)  [23,33].  This  set  may  be 
a  sequence  of  instructions  or  a  pattern  of  instructions.  Integrity  based  detection 
compares  a  known  good  entity  with  a  suspected  entity  in  order  to  verify  the  correct¬ 
ness/accurateness  of  the  suspect  entity.  An  example  of  such  is  comparing  “a  current 
snapshot  of  the  filesystem  or  memory  with  a  known,  trusted  baseline”  [11].  These 
two  techniques  within  the  static  detection  class  show  two  views,  that  of  detection  of 
known  bad  via  looking  for  the  bad  and  detection  of  bad  via  verifying  known  good. 

One  drawback  to  signature  detection  is  the  fact  that  it  works  via  the  blacklist 
(i.e.,  blocks  a  specific  list  of  disallowed  items,  ie, .  processes,  actions,  email  addresses) 
methodology  such  that  you  must  know  of  the  attack  before  blocking  it.  The  two 
limitations  cited  by  James  Foster  of  Global  Security  Solution  Development  and  the 
glossary  developed  by  Imperva,  Data  Security  for  the  Data  Center,  are:  1.  “They 
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are  prone  to  false  positives  without  extensive  tuning”,  and  2.  “They  are  not  effective 
at  detecting  many  unknown  attacks  on  custom  or  internally  developed  code”  [23, 
33].  However,  the  counterpart  to  blacklist  is  whitelist  (blocks  all  but  a  specific  list 
of  allowed  items,  ie,.  processes,  actions,  email  addresses,  etc,)  which  is  analogous 
to  integrity  based  detection.  By  including  both  signature  (analogy: blacklist)  and 
integrity  (analogy: whitelist)  in  the  static  detection  class  it  seems  that  we  have  covered 
static  analysis. 

It  is  our  hypothesis  that  rootkits  must  hide  from  both  classes  of  detection, 
namely  Behavioral  and  Static ,  in  order  to  be  completely  stealthy.  A  rootkit  does  not 
necessarily  need  to  hide  from  both  in  order  to  be  effective  but  in  order  to  obtain 
complete  stealth  it  must  address  both  and  be  at  the  lowest  ring/highest  level.  The 
term  “lowest  ring/highest  level”  refers  to  the  protection  rings  in  Figure  2.7.  If  we 
do  not  attempt  detection  in  all  classes  then  a  rootkit  could  achieve  stealthiness  by 
only  implementing  the  class  that  is  not  challenged.  For  example,  a  particular  rootkit 
could  use  metamorphic  code  in  order  to  subvert  signature  detection  and  hook  the 
integrity  scan  report  in  order  to  subvert  the  results  which  would  in  effect  defeat  the 
static  detection  class  which  shows  that  if  behavior  is  not  checked  then  we  will  not  be 
able  to  detect  such  a  rootkit. 

One  way  in  which  some  rootkits  can  be  found,  as  mentioned  by  Hoglund  et  al. 
is  by  looking  for  an  image  on  a  system,  which  falls  into  the  static  detection  class, 
“This  approach  is  still  used  by  most  anti-virus  vendors”  [8].  Hoglund  also  suggests 
that  “All  software  must  ‘live’  in  memory  somewhere”  [8].  which  means  that  detection 
can  also  be  implemented  in  the  forms  of  guarding  and  scanning. 

Guarding  vs  Scanning  -  Guarding  is  the  enumeration  and  protection  of  all  im¬ 
portant  “assets” .  This  can  be  compared  to  whitelists  (a  specific  list  of  allowed  items, 
ie,.  processes,  actions,  email  addresses,  etc,)  and  the  integrity  detection  technique. 
Guarding  then  disallows  anything  not  on  the  “approved”  whitelist.  Scanning,  on  the 
other  hand,  is  looking  for  a  particular  “signature”  among  the  unknown  events  and 
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possibly  within  the  known  events  (to  verify  the  known  events’  integrity).  Scanning 
requires  a  “signature”  which  assumes  apriori  knowledge  of  the  intrusion  or  of  a  par¬ 
ticular  attack  method  [8].  Scanning  could  then  be  compared  to  blacklists  and  the 
signature  technique  of  the  static  class. 

Both  guarding  and  scanning  are  legitimate  and  useful  techniques  to  protecting 
a  system  and  when  used  in  conjunction  with  one  another  will  raise  the  defenses  of  a 
given  system. 

Rootkit  techniques  can  be  used  to  detect  rootkits.  For  example,  the  usage  of  a 
rootkit  to  detect  other  rootkits.  If  we  use  a  rootkit  to  report  rather  than  to  hide  we 
may  be  able  to  subvert  a  malicious  rootkit  at  the  same  level  by  creating  a  somewhat 
“secure  channel”  for  reporting  back  to  the  user. 

The  lower  detour  function  box  in  Figure  3.13  (step  4),  shows  how  it  might  be 
possible  to  export  the  unmodified  data  of  the  target  program  to  a  different  desti¬ 
nation  in  order  to  maintain  data  integrity  and  compare  against  the  modified  source 
application  data,  creating  a  crossview.  Further  details  of  this  detection  technique  will 
be  discussed  in  Section  4.8. 

3.3.3  Rootkit  Detection  Examples  -  Behavioral. 

3.3.3. 1  VICE.  Vice  detects  the  presence  of  hidden  hooks  by  installing 
its  own  device  driver  to  check  the  SSDT  for  pointers  that  do  not  resolve  to  ntoskrnl.exe 
which  (“provides  the  Microkernel  and  Executive  layers  of  the  Windows  NT  kernel 
space,  and  is  responsible  for  various  system  services  such  as  hardware  virtualisation, 
process  and  memory  management”)  [67].  It  also  checks  devices  in  driver.ini  with 
the  IRP  table,  and  applications  looking  for  IAT  hooks  in  every  DLL.  This  method  of 
detection  is  very  robust  when  looking  for  hooks  of  this  type.  However,  Vice  is  subject 
to  subversion  through  other  techniques  such  as  DSM  [12], 
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Hoglund  et  al,  "Subverting  the  Windows  Kernel,  Rootkits"  2006. 

Figure  3.13:  Trampoline  Function  with  Modification  [8] 

3. 3. 3. 2  Patchfinder.  Patchfinder  is  a  proof  of  concept  tool  created 
by  Joanna  Rutkowska.  Patchfinder  works  by  puting  the  x86  processor  into  single 
step  mode  and  counting  every  instruction  on  a  clean  machine  and  comparing  the 
instruction  count  to  that  of  a  potentially  compromised  machine.  Through  statistics 
Rutkowska  claims  that  rootkit  detection  can  occur  because  the  application  of  a  his¬ 
togram  shows  shifts  in  peak  instruction  counts  on  an  infected  system  versus  a  non 
infected  system.  Differently  stated,  a  clean  system  will  have  a  peak  in  instruction 
counts  at  a  particular  spot  in  execution  and  an  infected  system  will  have  a  different 
or  shifted  peak.  However,  the  height  of  the  peak  may  vary,  normally,  because  of  the 
different  paths  of  execution  that  could  exist  [12]. 
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3. 3. 3. 3  System  Virginity  Verifier.  System  Virginity  Verifier,  created  as 
another  proof  of  concept  by  Joanna  Rutkowska,  works  much  like  VICE  by  “comparing 
important  system  libraries  and  drivers  on  disk  with  their  corresponding  loaded  images 
in  memory”  [12]. 

3. 3. 3. 4  Copilot.  Copilot  is  a  hardware  detection  tool  in  the  form  of  a 
PCI  card  which  implements  behavioral  and  signature  based  detection  techniques  on 
key  tables  and  functions.  Copilot  effectively  solves  (at  least  for  now)  the  need  to  have 
a  secure  channel  for  reporting  back  to  a  user  by  maintaining  all  of  the  processing  of 
information  on  its  own  CPU  using  Direct  Memory  Access  (DMA)  to  scan  for  rootkits, 
and  its  own  network  interface  to  securely  send  the  information  to  the  requestor  [12]. 

3.3.4  Rootkit  Detection  Examples  -  Signature. 

S.Sfi.  1  Klister.  Klister  finds  processes  by  iterating  through  the  sched¬ 
uler  to  find  all  threads  that  are  scheduled  and  comparing  that  list  to  a  higher  level 
view  of  running  processes  thus  allowing  a  crossview  to  find  discrepancies.  A  crossview 
uses  two  or  more  views  of  the  same  data  and  compares  the  results  in  order  to  identify 
discrepancies.  Klister  is  a  proof  of  concept  for  the  Windows  2000  platform  created  by 
Joanna  Rutkowska  [12].  It  is  claimed  to  be  subvertible  at  least  in  theory  by  changing 
the  scheduler  code  or  by  using  virtual  machine  technology.  However,  both  of  these 
subversion  techniques  not  only  raise  the  level  of  difficulty  for  rootkit  creation  but  have 
an  inherent  problem  in  that  they  increase  processor  workload,  thus  allowing  detection 
via  delay  issues. 

3. 3.4.2  Rootkit  Revealer.  Rootkit  revealer  seeks  to  find  persistent 
rootkits  (those  that  remain  between  reboots)  by  comparing  the  registry  hive  (“com¬ 
prises  a  set  of  files,  called  hives,  that  are  stored  on  the  hard  drive”  [41])  and  the 
filesystem  to  identify  files  that  do  not  belong  [12].  This  method  of  detection  can 
easily  be  subverted  through  redirection  of  queries  to  the  file  system  files  and  queries 
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to  the  registry.  According  to  the  technichal  paper,  An  Analysis  of  Forensic  Tools  in 
Detecting  Rootkits  and  Hidden  Processes  by  Todd  et  al  [3] ,  Rootkit  Revealer  was  able 
to  detect  AFXRootkit  and  Hacker  Defender. 

3.3.4 -3  Strider  GhostBuster.  Strider  Ghost  buster  seeks  to  find  rootk¬ 
its  by  comparing  high  level  API  calls  and  a  manually  parsed  version  of  the  filesystem 
to  identify  hies  that  do  not  belong  [12].  As  with  many  of  these  detection  techniques, 
this  method  of  detection  can  easily  be  subverted  through  use  of  hooking  and  redirec¬ 
tion.  Therefore,  it  is  important  that  the  information  obtained  is  verified  to  be  correct 
possibly  through  the  use  of  trusted  computing  methodology. 

3. 3-4 -4  Tripwire.  Tripwire  accomplishes  detection  through  integrity 
checking  of  the  disk.  I11  order  for  this  to  succeed  the  user  must  take  a  clean  view  of  their 
disk  prior  to  operation  and  then  any  subsequent  checks  are  compared  against  the  clean 
view.  Each  view  or  check  uses  a  Cyclic  Redundancy  Check  (CRC)  hash  value  to  record 
the  state  of  the  disk.  Any  variations  to  the  CRC  in  future  checks  from  the  original 
check  constitutes  a  flag  or  detection.  Subversion  of  this  technique  is  accomplished 
through  in  memory  rootkits  as  opposed  to  on  disk  (persistent)  rootkits  [12]. 

3. 3.4. 5  Blacklight.  Blacklight  works  by  querying  every  possible  Pro¬ 
cess  Identihcation  (PID)  number,  which  they  call  Process  Identihcation  Brute  Force 
(PIDB),  and  comparing  the  results  with  a  crossview  of  a  higher  level  call  to  mark 
any  discrepancies  as  hidden.  The  following  quote  from  an  article  published  on  un¬ 
informed. org  by  Peter  Silberman  and  C.H.A.O.S.  clearly  describes  how  Blacklight 
works: 


“Now  we  have  a  complete  picture  of  how  Blacklight  detects  hidden  pro¬ 
cesses:  Blacklight  starts  looping  through  the  range  of  valid  process  IDs, 
0  through  0x41DC.  Blacklight  calls  OpenProcess  on  every  possible  PID. 
OpenProcess  calls  NtOpenProcess.  NtOpenProcess  calls  PsLookupPro- 
cessByProcessId  to  verify  the  process  exists.  PsLookupProcessByProces- 
sld  uses  the  PspCidTable  to  verify  the  processes  exists.  NtOpenProcess 
calls  ObOpenObjectByPointer  to  get  the  handle  to  the  process.  If  Open- 
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Process  was  successful,  Blacklight  stores  the  information  about  the  process 
and  continues  to  loop.  Once  the  process  list  has  been  created  by  exhaust¬ 
ing  all  possible  PIDs.  Blacklight  compares  the  P1DB  list  with  the  list  it 
creates  by  calling  CreateToolhelp32Snapshot.  CreateToolhelp32Snapshot 
is  a  Win32  API  that  takes  a  snapshot  of  all  running  processes  on  the  sys¬ 
tem.  A  discrepancy  between  the  two  lists  implies  that  there  is  a  hidden 
process.  This  case  is  reported  by  Blacklight”  [63]. 

According  to  the  technichal  paper,  An  Analysis  of  Forensic  Tools  in  Detecting  Rootkits 
and  Hidden  Processes  by  Todd  et  al  [3],  BlackLight  was  able  to  detect  AFXRootkit, 
Hacker  Defender,  Vanquish,  FU  and  FUTo. 

3. 3. 4- 6  Ice  Sword.  Ice  Sword,  was  created  by  a  Chinese  programmer 
by  the  alias  of  pjL  [40].  It  is  believed  to  function  using  the  same  techniques  as 
Blacklight  for  process  detection  [63].  However,  an  apparent  advantage  Ice  Sword 
has  over  it’s  counterpart  is  that  it  is  more  robust  and  includes  capabilities  to  detect 
“hidden  processes,  services,  drivers,  hies,  ports,  and  registry  settings”  [3].  According 
to  the  technichal  paper,  An  Analysis  of  Forensic  Tools  in  Detecting  Rootkits  and 
Hidden  Processes  by  Todd  et  al  [3],  Ice  Sword  was  able  to  detect  AFXRootkit,  Hacker 
Defender,  Vanquish,  FU  and  FUTo. 

3. 3. 4- 7  GMER.  GMER  scans  for  hidden  processes,  threads,  modules, 
services,  hies,  alternate  data  streams,  registry  keys,  SSDT  hooks,  IDT  hooks,  IRP 
calls,  and  inline  hooks.  GMER  also  monitors  creation  of  processes,  driver  loading, 
library  loading,  hie  functions,  registry  entries,  and  TCP/IP  connections  [48]. 

The  area  of  rootkit  detection  has  grown  considerably  and  there  are  a  very  large 
number  of  detectors.  Due  to  the  high  number  of  detectors  available,  not  all  were  able 
to  be  covered  due  to  time  contraints.  However,  a  list  of  some  that  were  found  are 
listed  in  Table  3.3.  Some  were  found  mentioned  in  various  articles  but  not  described 
fully  and  some  were  found  at  sites  such  as  majorgeeks.com  [3]. 
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Table  3.3:  Rootkit  Detection  Examples 


Rootkit  Detector 

Static 

Behavioral 

Klister  (Rutkowska) 

X 

Rootkit  Revealer  (Sysinternals) 

X 

Stricter  Ghostbuster  (Microsoft) 

X 

Tripwire 

X 

F-Secure  Blacklight  (F-Secure) 

X 

Ice  Sword  (XFocus) 

X 

Patchfinder 

X 

Vice 

X 

System  Virginity  Verifier  (Rutkowska) 

X 

CoPilot 

X 

Other  Rootkit  Detectors  include:  RKDetector,  find  hidden  service,  Flister,  Kill 
hide  services,  Kernel  hidden  process/module  checker,  modGreper,  RegDatXP,  Task- 
Info,  and  bluestone. 

Windows  RK  Detectors  include:  Aries  Sony  Rootkit  Remover (lavasoft),  Archon 
Scanner (x-solve),  AVG  AntiRootkit(Grisoft),  Avira  Rootkit  Detection(Avira),  Dark 
Spy(CardMagic  and  Wowocock),  Helios(MIEL  e-Security),  HiddenFinder(Wenpoint), 
HookExplorer  (iDefense),  Panda  Anti-Rootkit  Tucan  (Panda  Software),  Process  Mas¬ 
ter  (Backfaces),  Rootkit  Detective  (McAfee  Avert  Labs),  Rootkit  Buster  (Trend  Mi¬ 
cro),  RootKit  Hook  Analyzer  (Resplendence),  RootkitShark  (Advances.com),  Rootkit 
Uncover  (Bit  Defender),  Rootkit  Unhooker  (UG  North),  SEEM  (Al,  nunki),  Sophos 
Antirootkit  (Sophos),  and  Unhackme  (Greatis). 

Linux/BSD  RK  Detectors  include:  chkrootkit(Murik)  and  Jessen),  Zeppoo  (Zep- 
poo),  and  Rootkit  Hunter  (Boelen). 

Mac  RK  Detectors  include:  OS  X  Rootkit  Hunter  (Christian  Hornung). 

Summary  -  The  detection  classes,  techniques,  and  implementations  with  their 
associated  examples  all  give  us  a  better  view  into  how  rootkits  can  be  detected  and 
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where  we  may  find  potential  deficiencies.  The  following  chapter  will  show  attack 
trees  and  defense  trees  as  examples  of  how  rootkits  hide  and  are  detected  in  order  to 
help  further  identify  what  these  deficiencies  may  be.  It  also  illustrates  via  ideas  and 
experimentation  some  ways  to  further  the  state  of  the  art. 
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IV.  Experimentation  and  Results 


4-1  Chapter  Overview 

In  this  chapter,  in  order  to  further  explain  and  classify  rootkit  hiding  techniques, 
we  develop  an  attack  tree  and  a  defense  tree  which  will  help  to  identify  deficiencies 
in  current  detection,  help  focus  future  research  of  hiding  and  detection,  as  well  as 
important  concepts  for  defense.  We  also  explain  and  define  an  experiment  which 
shows  how  a  rootkit  can  be  used  to  defend  against  other  rootkits  of  the  same  type  in 
order  to  springboard  future  defensive  techniques. 

4-2  Rootkit  Attack  Tree 

In  Figure  4.1  we  have  created  an  example  of  how  an  attacker  might  achieve 
a  degree  of  stealth  via  following  one  of  the  branches  of  this  attack  tree  through  its 
associated  OR  gates.  We  note  that  in  order  to  obtain  a  degree  of  stealth  the  attack 
need  only  pick  one  of  the  shown  attack  paths.  For  example,  the  FU  rootkit  [26] 
modifies  the  SSDT  which  provides  some  stealth  because  the  SSDT  is  relied  upon  by 
many  programs  to  provide  information  about  existing  processes. 


AttackTree  for  Rootkit  Hiding 

_ Rootkit  Technology  Timeline _ 

Future  -  Bleeding  Edge  --  Cutting  Edge - Current . Older 


Figure  4.1:  Attack  Tree  2:  Rootkit  Hiding  Techniques 
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From  Figure  4.1  a  hierarchical  view  of  complete  stealth  can  be  derived  and  is 
shown  in  Figure  4.2.  Thus,  for  complete  stealth  an  attacker  must  descend  to  the 
lowest  level  of  stealth.  However,  the  lowest  level  of  stealth  needed  is  dictated  by 
the  level  of  defense  being  implemented.  Furthermore,  descending  to  the  lowest  level 
without  regard  to  the  higher  levels  is  also  not  sufficient;  the  attacker  must  keep  in  mind 
the  30,000  foot  view  in  order  to  maintain  stealthiness.  For  example,  if  an  attacker 
descends  to  the  lowest  level  of  stealth  (perhaps  using  a  VM  technique  such  as  Blue 
Pill)  but  forgets  to  hide  from  both  the  signature  and  behavioral  detection  classes 
with  all  of  their  techniques  and  implementations  then  they  can  still  be  theoretically 
detected.  In  this  case,  although  popular  belief  and  albeit,  very  good  analysis,  would 
suggest  that  Blue  Pill  would  be  “...virtually  ‘100%  undetectable’!”  [54],  this  does  not 
account  for  static  or  behavioral  analysis  from  a  hardware  implementation  and  possibly 
others. 
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Threads 


VM 
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Figure  4.2:  Rootkit  Hierarchy 
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4-3  Rootkit  Defense  Tree 


A  defense  tree  is  the  counterpart  to  an  attack  tree,  and  is  used  simply  to  identify 
what  needs  to  be  defended  in  a  computer  system.  In  order  to  have  complete  protec¬ 
tion,  all  of  the  branches  of  the  tree  must  be  checked  as  can  be  seen  in  Figure  4.3  which 
uses  AND  gates  to  depict  that  all  branches  must  be  addressed. 


Defense  Tree 


Rootkit  Detection  Technology  Timeline _ 

Bleeding  Edge  --  Cutting  Edge . Current . Older  | 


Figure  4.3:  Rootkit  Defense  Tree 

However,  some  branches  can  potentially  be  checked  via  checking  other  branches 
or  combinations  of  branches.  An  example  of  defense  tree  usage  is  given  in  Figure  4.4, 
with  Klister.  By  using  a  view  of  the  API  branch  and  a  view  of  the  scheduler/ dispatcher 
(the  part  of  the  operating  system  that  decides  what  is  to  run  next  and  for  how  long 
based  on  priorities),  Klister  is  able  to  effectively  discover  what  is  occuring  in  one 
branch  without  directly  searching  it  because  the  dispatcher  should  hold  an  accurate 
list  of  what  is  running  which  can  be  compared  against  what  “should  be”  accurate 
in  the  API  list.  Any  discrepancies  can  then  be  shown  as  hidden  processes.  This 
technique  cuts  out  the  DKOM  technique  used  by  FU  and  FUTo  by  working  around 
it.  However,  as  mentioned  earlier,  all  branches  should  still  be  checked  because  if  the 
rootkit  gets  below  the  dispatcher  then  this  technique,  as  clever  as  it  is,  also  fails. 
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Figure  4.4:  Rootkit  Defense  Tree:Klister  Example 

4-4  Furthering  the  state  of  the  art 

In  order  to  increase  the  state  of  the  art  in  finding  techniques  we  need  to  continue 
to  innovate  and  create  new  ways  to  detect  and  “win”  the  arms  race.  One  possible 
way  to  increase  detection  abilities  is  to  use  the  available  hiding  abilities.  To  this  end, 
we  have  created  the  following  experiment  which  uses  a  hooking  rootkit  to  find  other 
hooking  rootkits.  In  Section  4.5  we  outline  the  System  Under  Test  (SUT),  which  is 
then  followed  by  an  experiment  overview  in  Section  4.6,  setup  in  Section  4.7,  and 
outcomes  in  Section  4.8. 

4-5  System  Under  Test 

The  system  boundaries  for  this  particular  research  must  be  limited  in  order  to 
be  manageable  and  contribute  information  to  the  body  of  knowledge  on  rootkits.  For 
this  research  we  make  the  following  boundaries: 
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4-5.1  Base  System.  Dell  Latitude  D510,  Intel  Celeron(R)M  1.40GHz  pro¬ 
cessor,  1G  Ram,  Windows  XP  Professional  version  2002;  service  pack  2;  fully  patched 
as  of  8/21/2006. 

4-5.2  Tested  System.  VMware  Workstation  version  5.5.1  buildl9175  with 
Windows  XP  Professional  version  2002;  service  pack  2;  unpatched. 

4-5.3  Software.  Rootkits:  HideProcessHookMDL,  Modified  HideProcessHook- 
MDL,  HxdeflOOr,  AFX2005,  and  Fu.  Support  Software:  Procexp.exe,  dbgview.exe, 
calc.exe 

4-6  Experiment  Overview:  Rooted  Rootkits 

It  is  our  hypothesis  that  if  a  rootkit  hides  using  a  particular  method  that  it  can 
also  be  modified  to  find  other  rootkits,  of  the  same  or  “lesser”  classes,  using  that  same 
method.  The  first  experiment  simply  checks  to  see  if  a  rootkit  “HideProcessHook¬ 
MDL”  can  be  used  to  detect  other  rootkits  that  use  the  same  and  similar  methods  of 
hiding. 

HideProcessHookMDL  (HPH)  hides  by  hooking  the  SSDT  and  scrubbing  the 
content  of  the  returned  linked  list  for  anything  that  has  _root_  in  its  name  [25]. 

To  detect  the  ability  to  find  other  rootkits  we  modified  HideProcessHookMDL 
such  that  it  still  hooks  the  SSDT  but  does  not  hide  any  processes,  rather,  it  sends  a 
debug  print  statement  which  can  be  seen  using  Dbgview.exe  [51]  from  Sysinternals. 
Due  to  the  technique  used  to  hide  via  a  hook  into  the  SSDT,  nothing  is  readily 
seen  as  output  until  a  call  is  generated  looking  for  processes.  Therefore,  in  order 
to  more  closely  monitor  and  show  results  for  this  experiment  we  used  another  tool 
called  procexp.exe  [52]  also  from  Sysinternals,  because  it  continually  calls  for  a  list  of 
running  processes.  This  also  gives  us  the  ability  to  see  which  processes  are  running 
from  the  application  perspective  (procexp.exe)  so  that  we  can  compare  those  with 
what  the  modified  hook  (via  the  debug  statements  in  dbgview.exe)  sees  running.  It 
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is  also  necessary  to  name  a  program  with  jroot_  in  the  name.  For  these  experiments 
we  will  simply  make  a  copy  of  calc.exe  and  name  it  jroot_calc.exe.  The  final  piece 
of  software  needed  to  make  this  experiment  complete  is  InstDriver.exe  [29]  which  is 
used  to  install  the  modified  HPH. 

4-1  Experiment  setup 

Install  Modified  HPH  on  a  clean  system  (Windows  XP  used).  Start  HPH.  Start 
Debugview.  Start  Process  explorer.  Start  calc.exe  and  root  calc.exe.  Install  and 
start  other  hooks.  Compare  output  of  application  and  modified  hook. 

4-8  Outcomes 

4-8.1  Experiment  1  Modified  HPH  vs  unmodified  HPH.  This  experiment 
verified  that  HPH  was  working  because  the  process  (_root_calc.exe)  did  not  show  up 
in  the  process  explorer.  However,  it  showed  that  this  particular  hook  could  detect 
itself  because  the  hook  did  show  up  in  debugview. 

4-8.2  Experiment  2  Modified  HPH  vs  HxdeflOOr.  This  experiment  verified 
that  both  HPH  and  hxdeflOOr  were  working  because  the  process  (hxdefcalc.exe)  did 
not  show  up  in  the  process  explorer.  However,  it  did  show  up  in  debugview.  This 
illustrates  that  the  modified  HPH  was  able  to  defeat  hxdeflOOr. 

4-8.3  Experiment  3  Modified  HPH  vs  AFX2005.  This  experiment  verified 
that  both  HPH  and  AFX2005  were  working  because  the  process  (_root_calc.exe)  did 
not  show  up  in  the  process  explorer.  However,  it  did  show  up  in  debugview.  Which 
furthermore  illustrates  that  the  modified  HPH  was  able  to  defeat  AFX2005. 

4.8.4  Experiment  4  Modified  HPH  vs  Fu.  As  expected  this  experiment 
showed  that  Fu  was  able  to  continue  hiding  processes  even  with  the  modified  HPH 
installed  because  Fu  uses  a  different  hiding  technique  discussed  previously  called 
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DKOM.  This  experiment  further  illustrates  the  “arms  race”  that  exists  with  rootkits 
versus  rootkit  detectors. 

4-9  Metrics 

Metrics  in  this  experiment  only  had  two  outcomes,  success  or  failure.  The  results 
are  summarized  in  Figure  4.5. 

TEST  RESULTS 


Modified  HPH  vs  ... 

Success 

Failure 

Unmodified  HPH 

X 

HxdeflOOr 

X 

AFX2005 

X 

FU 

X 

Figure  4.5:  Test  Results 


In  this  chapter  we  have  shown  how  attack  trees  and  defense  trees  can  be  devel¬ 
oped  in  order  to  identify  holes  in  detection  and  also  ways  to  detect  in  all  branches 
even  when  a  particular  branch  has  been  otherwise  compromised. 
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V.  Summary 


5.1  Conclusion 

This  thesis  has  shown  the  importance  of  understanding  rootkit  stealth  tech¬ 
niques  and  rootkit  detection  techniques.  Many  of  the  current  technologies  for  each 
stealth  technique  and  detection  class  were  illustrated  in  Chapter  3.  Rootkit  stealth 
techniques  can  be  examined  using  a  graphical  representation  called  an  attack  tree. 
The  attack  tree  and  its  counterpart  the  defense  tree  developed  in  this  thesis  show  what 
categories  of  techniques  touch  the  base  of  the  tree  and  shows  to  successfully  defend 
against  all  rootkits,  each  and  every  category  of  rootkits  must  be  at  least  addressed.  It 
is  also  important  to  remember  that  this  is  rootkit  research  is  very  comparable  to  an 
arms  race  in  that,  what  works  today  may  not  work  tomorrow  because  the  attackers 
will  invent  new  ways  of  getting  in  that  we  as  defenders  have  not  yet  thought  of  and 
visa  versa. 

It  was  experimentally  shown  that  a  rootkit  can  successfully  find  other  rootkits. 
This  research  explored  ways  in  which  rootkits  hide  and  also  how  they  can  be  detected. 
These  hiding  and  finding  techniques  were  used  to  create  an  attack  tree  from  which 
we  can  identify  deficiencies  in  current  detection  techniques.  Detection  categorizations 
were  also  refined. 

The  following  section  outlines  some  ideas  for  future  research. 

5.2  Future  Research 

5.2.1  Rootkit  Detection  concept:  Screen  Sweeping.  Screen  Sweeping  is  re¬ 
moving  data  and/or  results  from  the  computer  system  user’s  view.  If  a  rootkit  is 
installed  on  a  machine  and  a  rootkit  detector  successfully  identifies  that  there  is  a 
rootkit  present,  the  next  logical  step  is  to  alert  the  user.  If  a  rootkit  can  stop  or 
modify  the  detection  report  before  it  is  sent  to  the  computer  screen  then  the  user  will 
not  know  that  a  rootkit  is  present  even  if  the  detection  tool  was  initially  successful. 

Screen  Sweeping ,  suggests  that  in  order  to  detect  perfectly  we  must  not  only 
detect  but  create  a  “secure  channel”  from  the  level  of  detection  to  the  screen.  This 
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“secure  channel”  or  “Secure  I/O”  [38]  is  one  of  the  concepts  that  is  currently  being 
addressed  by  work  in  Trusted  Computing,  which  is  explained  in  Section  2.4. 

5.2.2  Rootkit  Detection  concept:  Memory  Tracking.  Every  system  has  a 
finite  amount  of  memory.  We  understand  how  memory  is  used  and  the  algorithm¬ 
s/applications  that  choose  which  memory  will  be  used.  If  we  can  mark  or  track  which 
memory  is  being  used  by  our  known  system  and  which  memory  is  simply  “trash”  then 
by  carefully  tracking  the  memory  we  should  be  able  to  see  writing  anomalies  and  thus 
detect  other  potentially  malicious  software  on  our  systems.  For  example,  if  we  know 
that  we  have  100  memory  locations  in  our  system  and  locations  1-10  are  being  used, 
then  we  should  know  that  upon  installation  of  a  new  program  or  other  such  action 
that  memory  location  11  should  be  written  used  (location  of  the  next  memory  lo¬ 
cation  would  be  algorithm/software  dependent  but  should  be  understood).  If  upon 
installation  we  tracked  memory  and  noted  that  the  chosen  location  for  installation 
was  not  11  but  rather  some  other  location  then  we  can  set  a  flag  as  anomalous  and 
further  investigate.  What  is  at  memory  location  11?  Is  it  a  bad  sector?  Is  it  malicious 
code? 


5.2.3  Research  Questions. 

1.  Do  multiple  detection  methods  (ie,  crossview)  need  to  be  used  or  can  a  trusted 
channel  be  created  that  is  sufficient  to  detect  and  report  all  rootkits? 

2.  Is  the  first  “complete”  rootkit  installed  on  a  machine  truly  the  winner?  Is  there 
no  other  detection/removal  option  if  a  rootkit  addresses  each  category  and  each 
level? 

3.  Can  I  detect  hardware  scans  and  feed  them  faulty  information  or  hide  from 
them? 

4.  Are  there  any  other  classes  of  detection  other  than  static  and  behavioral? 

5.  Are  there  any  other  implementations  of  the  static  detection  class  other  than 
signature  and  integrity? 
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6.  Are  there  any  other  implementations  of  the  behavioral  detection  class  other 
than  anomaly  and  signature? 

7.  Are  there  any  other  detection  techniques  other  than  crossview,  hardware,  cog¬ 
nitive/human,  software,  remote  attestation  and  memory  tracking? 

8.  Explore  ways  to  implement  memory  tracking. 
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Appendix  A.  Windows  Architecture 
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Figure  A.l:  Windows  Architecture  [19] 
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Appendix  B.  UNIX  Architecture 


Unix  Functional  Layer  Model 


Figure  B.l:  UNIX  Architecture  [24] 
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Figure  B.2:  UNIX  Architecture  [24] 
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Glossary 


Rootkit 

Tools  designed  to  create  and  maintain  an  environment 
on  a  computer  in  which  attack  tools  and  activities  may 

be  hidden,  such  that  a  user  does  not  know  of  their 
presence  on  a  compromised  machine., 

Signature  based  detection 

Data  is  scanned  for  a  pattern  that  comprise  a  “finger¬ 
print”  that  is  unique  to  a  particular  entity  [11]., 

Heuristic/behavioral  based  detection 

Behavioral  based  detection  seeks  to  identify  actions  or 
patterns  that  are  abnormal  to  system  operation.  These 
detection  techniques  “work  by  recognizing  deviations  in 
“normal”  system  patterns  or  behaviors”  [11]., 

Crossview  based  detection 

Crossview  based  detection  uses  the  idea  of  data  redun¬ 
dancy  (the  existence  of  equivalent  data  in  more  than 
one  location)  to  find  “answers”  from  multiple  sources 
that  should  be  the  same  in  order  to  identify  discrepan¬ 
cies.  An  example  of  such  is  a  child  asking  her  mother 
for  permission  to  go  to  the  park  and  then  asking  her 
father  permission  for  the  same  thing.  The  two  answers 
should  be  the  same;  however,  the  difference  in  answers 
is  what  is  exploited  by  the  child.  Detection  techniques 
would  use  the  difference  in  answers  to  identify  the  ex¬ 
istence  of  a  rootkit  [11]., 

Integrity  detection 

Integrity  based  detection  compares  a  known  good  en¬ 
tity  with  a  suspected  entity  in  order  to  verify  the  cor¬ 
rectness/accurateness  of  the  suspect  entity.  An  ex¬ 
ample  of  such  is  comparing  “a  current  snapshot  of 
the  filesystem  or  memory  with  a  known,  trusted  base¬ 
line”  [11]., 
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Hardware  based 


Steganography 


detection  Hardware  based  detection  uses  a  piece  of  hardware  to 

implement  signature,  heuristic,  crossview,  or  integrity 
based  detection  while  separating  itself  from  the  sus¬ 
pected  operating  system  [11]., 

Steganography  as  quoted  from  the  Mirriam- Webster’s 
online  dictionary  is  “the  art  or  practice  of  concealing  a 
message,  image,  or  hie  within  another  message,  image, 
or  hie”  [45]  However,  a  short  definition  which  describes 
how  steganography  works  is:  hiding  in  plain  sight., 
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